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1  Overview 


The  Integrated  Cognitive-neuroscience  Architectures  for  Understanding  Sensemaking  (ICArUS)  Program 
aimed  to  build  computational  cognitive-neuroscience  models  to  explain,  predict,  and  emulate  the 
process  of  human  sensemaking  -  the  process  of  generating  hypotheses  to  explain  data  that  is  sparse, 
noisy,  and  uncertain.  Phase  1  lasted  from  December,  2010  to  December,  2012.  In  Phase  1,  the 
program  employed  a  challenge  problem,  based  on  geospatial  data,  to  compare  the  models' 
performance  to  human  performance.  Here,  the  term  geospatial  data  includes  multiple  types  of 
intelligence  data,  any  of  which  may  be  represented  in  a  georeferenced  framework  -  for  instance,  a 
layer  within  a  GIS-like  display.  The  challenge  problem  was  focused  on  scientific  rigor  as  opposed  to 
operational  relevance;  all  examples  were  fictitious  in  nature  and  represented  in  a  neutral  manner. 

This  document's  primary  purpose  is  to  specify  the  format  of  the  ICArUS  Phase  1  Challenge  Problem.  It 
was  written  for  software  developers,  contains  development  code,  and  is  not  meant  to  stand  alone. 

The  document,  ICArUS:  Overview  of  Test  and  Evaluation  (T&E)  Materials,  available  at 
http://www.mitre.orR/publications,  provides  a  broad  overview  of  MITRE's  T&E  effort,  including  a  list  of 
all  T&E  documents  and  materials  produced  in  Phase  1. 


All  content  following  this  page  is  copied  verbatim  from  the  original  report  prepared  in  May,  2012. 
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2  Introduction 


Consistent  with  the  ICArUS  Broad  Agency  Announcement  (BAA),  all  tasks  of  the  Phase  1  Challenge 
Problem  involve  spatial  context  frames  with  underlying  probabilities  that  are  constant  in  time.  Tables 
1  and  2  outline  the  7  tasks  that  comprise  the  Challenge  Problem  and  the  probabilistic  if-then  rules 
(PROBS)  that  govern  each  task  respectively.  The  primary  distinctions  among  the  tasks  are  as  follows: 
Tasks  1-3  require  statistical  learning  of  spatial  frames  -  i.e.,  evidence  is  accumulated  over  time  from 
reports  of  individual  attacks;  Tasks  4-6  require  rule-based  sensemaking  and  integration  of  evidence 
(INTs)  over  layers  -  i.e.,  evidence  about  single  attacks  are  provided  by  multiple  independent  sources 
coincident  in  time.  Task  7  combines  aspects  of  earlier  tasks  while  introducing  self-initiated 
sensemaking. 

This  document  focuses  on  5  types  of  intelligence  data  (INT),  including: 


HUMINT  (Human  Intelligence):  intelligence  derived  from  information  collected  by  human  sources. 

IMINT  (Image  Intelligence):  intelligence  derived  from  buildings,  roads,  terrain  elements,  or  other 
objects  or  physical  features  of  the  environment. 

MOVINT  (Movement  Intelligence):  intelligence  derived  from  moving  things,  e.g.,  traffic  density. 

SIGINT  (Signals  Intelligence):  intelligence  gathered  by  the  interception  of  signals,  e.g.,  a  phone  call. 

SOCINT  (Sociocultural  Intelligence):  intelligence  based  on  people  and  their  communities,  including 
customs,  moral  attitudes,  and  culture. 

Since  the  ICArUS  model  will  not  be  required  to  process  raw  imagery  or  perform  low-level  feature 
extraction,  all  geospatial  data  (scene  objects)  are  represented  as  a  multidimensional  feature  vector. 
This  document  specifies  the  format  of  that  vector  as  well  as  the  format  of  the  ICArUS  models'  output. 
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Table  1:  Description  of  7  Tasks 


Task 


1 


Stimuli 


Description 


Learning  two  groups.  [10  blocks  of  10  trials.]  Attacks  by  two 
groups  (X,  O)  are  generated,  one  on  each  trial,  by  two  normal 
distributions  -  each  with  a  different  center  location  and 
dispersion.  On  the  last  trial  of  each  block  the  subject  is  shown 
an  attack  location  and  asked  to  predict  the  probability  of  attack 
by  each  group,  {X%,  0%}.  The  subject  then  makes  a  forced 
choice  (X  or  O).  At  the  end  of  the  trial  the  subject  sees  ground 
truth  (X  or  O)  at  the  attack  location  and  he  reports  surprise  (on 
7-point  Likert  scale). 


2 


Learning  four  groups.  [10  blocks  of  20  trials]  Unlike  Task  1, 
there  are  four  different  groups  (A,  B,  C,  D).  Also,  after 
reporting  {A%,  B%,  C%,  D%},  the  subject  is  required  to  draw 
four  circles  -  each  representing  the  “two-to-one  boundary”  for 
a  group  (i.e.,  attack  by  a  group  is  twice  as  likely  inside  the 
circle  compared  to  outside  the  circle).  At  the  end  of  the  trial  the 
subject  is  shown  ground  truth  (A,  B,  C,  or  D)  at  the  attack 
location  and  he  reports  surprise. 


3 


Finding  centers.  [10  blocks  of  20  trials]  Unlike  Task  2,  the 
task  is  to  find  group  centers  along  roads  “as  the  cow  walks” 
(not  as  the  crow  flies).  After  each  block  of  trials,  the  subject 
reports  {A%,  B%,  C%,  D%}  at  the  attack  location.  The  subject 
then  clicks  four  points  A,  B,  C,  and  D  on  the  roads  -  each 
representing  the  “center  of  gravity”  for  a  group  (i.e.,  maximum 
likelihood  location  for  a  group’s  center,  given  the  group’s 
attacks).  At  the  end  of  the  trial  the  subject  is  shown  ground 
truth  (A,  B,  C,  or  D)  at  the  attack  location  and  he  reports 
surprise. 
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Table 

Task 


(Continued):  Description  of  7  Tasks 

Stimuli 


Description 


4 


5 


Scoping  attacks.  [10  trials]  Unlike  Task  3,  the  subject  is 
given  the  center  (from  HUMINT)  for  one  group  A,  along  with 
four  possible  locations  1,  2,  3,  and  4  of  attack  by  that  group. 
The  subject  is  required  to  estimate  the  probability  of  attack  at 
each  location  based  on  a  normal  distance  decay  function 
along  roads.  The  subject  is  then  given  the  regional  boundaries 
for  groups  (SOCINT),  along  with  the  inside/outside  region 
attack  likelihoods.  The  subject  is  required  to  update  the  prior 
(HUMINT)  probabilities.  The  subject  then  allocates  resources 
in  %  across  the  sites.  At  the  end  of  the  trial  the  subject  is 
shown  ground  truth  (1,  2,  3,  or  4).  The  subject  reports  surprise 
after  each  datum. 


Fusing  layers.  [10  trials  with  4  stages]  Unlike  Task  4,  the 
subject  is  given  an  attack  location  (SIGACT)  along  with 
group  centers  (HUMINT)  A,  B,  C,  and  D  for  four  groups.  The 
subject  reports  probabilities  of  attack  by  each  group  { A%, 
B%,  C%,  D%}.  The  subject  is  then  given  four  more  INTS 
(IMINT,  MO  VINT,  SIGINT,  SOCINT)  in  four  stages,  along 
with  likelihoods  of  these  data  for  each  group.  At  each  stage 
the  subject  reports  surprise  and  updates  probabilities  { A%, 
B%,  C%,  D%}.  After  the  last  stage  the  subject  allocates 
resources  in  percentages  {A%,  B%,  C%,  D%}  against  the 
groups.  At  the  end  of  the  trial  the  subject  is  shown  ground 
truth  and  he  reports  surprise. 
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Choosing  layers.  [20  trials  with  6* *  stages]  Much  like  Task  5, 
except  the  subject  decides  which  layer  of  INT  (IMINT, 

MO  VINT,  SIGINT,  SOCINT)  to  receive  at  each  stage  of  each 
trial.  The  subject  chooses  a  total  of  three  layers  in  any  order. 
The  criterion  for  each  choice  is  to  maximize  the  amount  of 
information  gain  expected  from  the  INT. 

*Each  INT  is  counted  as  two  stages  of  the  task  because  it 
requires  a  decision  (choosing  a  layer)  and  an  inference  (of 
group  probabilities). 
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Table  1  (Continued):  Description  of  7  Tasks 


Task 


Stimuli 


Description 


Self-initiated  sensemaking.  [15  trials].  Task  7  is 
a  combination,  in  part,  of  Tasks  4  and  6,  but  with 
a  focus  on  self-initiated  sensemaking. 

Participants  will  be  given  the  centers  of  four 
groups  (A-D)  and  four  possible  attack  sites  (1- 
4).  As  in  Task  4,  the  primary  task  is  to  predict 
the  location  of  attack  and  to  allocate  resources  to 
each  location  {@1%,  @2%,  @3%,  @4%}.  But 
unlike  Task  4,  participants  will  decide  if  and 
when  to  seek  additional  data,  as  well  as  which 
data  (INT)  to  seek.  See  the  Phase  1  Challenge 
Problem  Design  and  Test  Specification  for  a  more 
detailed  description  of  Task  7,  including  a 
summary  of  the  major  structural  differences 
between  Task  7  and  Tasks  1-6. 
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Table  2:  Probabilistic  If-then  Rules  (PROBS)  for  each  INTType 


INTS 

PROBS 

HUMINT 

If  a  group  attacks,  then  the  likelihood  of  attack  is  a  Gaussian  function  (a  =  10  miles) 
of  distance  along  road(s)  from  the  group's  center. 

If  the  attack  is  near  a  government  building,  then  attack  by  A  or  B  is  four  times  as 
likely  as  attack  by  C  or  D. 

IMINT 

If  the  attack  is  near  a  military  building,  then  attack  by  C  or  D  is  four  times  as  likely 
as  attack  by  A  or  B. 

MOVINT 

If  the  attack  is  in  dense  traffic,  then  attack  by  A  or  C  is  four  times  as  likely  as  attack 
by  B  or  D. 

If  the  attack  is  in  sparse  traffic,  then  attack  by  B  or  D  is  four  times  as  likely  as  attack 
by  A  or  C. 

SIGINT 

If  SIGINT  on  a  group  reports  chatter,  then  attack  by  that  group  is  seven  times  as 
likely  as  attack  by  each  other  group. 

If  SIGINT  on  a  group  reports  silence,  then  attack  by  that  group  is  one-third  as  likely 
as  attack  by  each  other  group. 

SOCINT 

If  the  attack  is  in  a  group's  region,  then  attack  by  that  group  is  twice  as  likely  as 
attack  by  each  other  group. 
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3  Input  Format 


The  Phase  1  Challenge  Problem  is  implemented  in  7  tasks.  Each  task  is  specified  by  an  ExamPhase 
element  in  a  XML  input  file,  and  each  phase  is  comprised  of  one  or  more  trials  (see  Figures  1  and  2). 
For  a  primer  on  XML,  see  http://www.w3schools.com/web/web  xml.asp.  An  XML  schema  defining  all 
XML  elements  described  in  this  guide  is  packaged  with  the  challenge  problem  software,  available  at 
http://www.mitre.org/research/technology-transfer. 

Each  exam  begins  with  an  lcarusEvaluation_CPDl  tag;  an  id  attribute  provides  a  unique  identifier  for 
the  exam,  and  an  exomTimeStamp  attribute  (optional)  indicates  when  the  exam  file  was  last  modified 
A  GridSize  element  specifies  the  size,  scale,  and  geographic  location  (latitude  and  longitude)  of  the 
scene.  Attributes  include  gridWidth,  gridHeight,  milesPerGridUnit,  bottomLeftLat,  and  bottomLeftLon 
This  information  is  used  to  convert  between  a  geographic  (latitude  and  longitude)  and  scene-based 
coordinate  system.  Coordinate  translation  and  specification  are  described  in  more  detail  in  Sections 
3.8-3.10. 

Next,  one  or  more  ExamPhase  elements  specify  the  tasks  contained  in  the  exam.  Each  ExamPhase 
element  has  both  an  exomld  attribute  (that  specifies  the  exam  it  is  part  of)  and  an  id  attribute  (that 
provides  a  unique  identifier  for  the  exam  phase).  Appendix  A  provides  an  entire  sample  exam. 


<?xml  version ="1.1"  encodings "UTF-8 "  standalone^ "yes "?> 

<ns3 : IcarusEvaluation  CPD1  xmlns : ns2= "IcarusCPD  Base" 

xmlns : ns3= "IcarusCPD  1 "  xmlns : xsi= "http : //www. w3 . org/2001 /XMLSchema- 
instance " 

examTimeStamp= "201 1-07 -27T12 : 31 : 54 . 989-04 : 00"  name= "Sample  Exam " 
id = "Sample  Exam"> 

<!--  Grid  size,  scale,  and  location  information  --> 
<GridSize  gridWidth= "1 00 "  qridEeiqht="100"  milesPerGridUnit= 
bottomLeftLat ="0. 0"  bottomLef tLon= "0 . 0" /> 

"0.2" 

<!--  Task  1  --> 

<ExamPhase  xsi : type= "ns3 : Task  1  Phase"  examld =" Sample  Exam" 
name=  "Mission  1 "  id  =”Taskl">  ... 

</ExamPhase> 

<!--  Task  2  --> 

<ExamPhase  xsi : type= "ns3 : Task  2  Phase"  examld =" Sample  Exam" 
name=  r'  Mission  2"  id  ="Task2  ">  ... 

</ExamPhase> 

<!--  Task  3  --> 

<ExamPhase  xsi : type= "ns3 : Task  3  Phase"  examld =" Sample  Exam" 
name= "  Mission  3"  id  ="Task3">  ... 

</ExamPhase> 

<!--  Task  4  --> 

<ExamPhase  xsi : type= "ns3 : Task  4  Phase"  examld =" Sample  Exam" 
name= "  Mission  4"  id  ="Task4">  ... 

</ExamPhase> 
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<!--  Task  5  — > 

<ExamPhase  xsi : type= "ns3 : Task_5_Phase ”  examl d= "Sample  Exam" 
name=  Mission  5"  id  =  "Task5  ">  ... 

</ExamPhase> 

<!--  Task  6  — > 

<ExamPhase  xsi  :  type=  "ns3 :  Task_6_Phase  ”  examl d=  "Sample  Exam" 
name= Mission  6"  id  ="Task6">  ... 

</ExamPhase> 

<!--  Task  7  — > 

<ExamPhase  xsi : type= "ns3 : Task_ 7_Phase "  examld =” Sample  Exam" 
name=  Mission  7"  id=  ” Task  7  ff>  ... 

</ExamPhase> 

</ns3 : IcarusEvaluation_CPDl> 

Figure  1:  Overview  of  an  exam  document 


3.1  Task  1:  Learning  two  groups 


For  convenience,  the  following  task  summary  is  reproduced  here  from  Table  1. 

Task  summary:  Learning  two  groups.  [10  blocks  of  10  trials]  Attacks  by  two  groups  (A,  B)  are 
generated,  one  on  each  trial,  by  two  normal  distributions  -  each  with  a  different  base  rate,  center 
location,  and  dispersion.  On  the  last  trial  of  each  block  the  subject  is  shown  an  attack  location  and 
asked  to  predict  the  probability  of  attack  by  each  group,  {A%,  B%}.  The  subject  then  makes  a  forced 
choice  (A  or  B),  to  allocate  resources  against  the  attack.  At  the  end  of  the  trial  the  subject  sees  ground 
truth  (A  or  B)  at  the  attack  location  and  reports  surprise  (on  7-point  Likert  scale). 

Task  1  is  defined  by  an  ExamPhase  element  of  type  Task_l_Phase.  The  task  is  divided  into  10  blocks, 
each  defined  by  a  TrialBlock  element  of  type  Task_l_TrialBlock;  each  block  contains  10  trials.  Each 
TrialBlock  element  also  contains  a  trialBlockNum  attribute  that  specifies  the  block  number,  and  a 
numPresentationTrials  attribute  that  specifies  the  number  of  training  trials  during  which  attacks  are 
shown.  Note:  no  response  is  expected  during  these  trials. 


<!--  Task  1  --> 

<ExamPhase  xsiityp e="ns3 : Task_l_Phase"  examld =" Sample  Exam" 
name=  "Mission  1"  id  ="Taskl"> 

<TrialBlock  xsi : type= "ns3 : Task_l_TrialBlock "  trialBlockNum= "1 " 
numPresentationTrials^ "9 "> 

<FeatureVectorFile  f eatureVectorUrl_CSV= " taskl_l . csv" 
f eatureVectorUrl_KML= ”taskl_l . kml ”/ > 

<ProbeTrial  trialNum ="10"> 

<GroupProbe> 

<AttackLocation  locationld ="1"  x="25"  y ="18" 
lat=” 0 .13025057227020484"  lon= "0 . 1 809035 7259 75067 "/> 
<Groups>A  B</Groups> 

</ GroupProbe> 

<TroopSelectionProbe> 
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<Groups>A  B</Groups> 

</TroopSelectionProbe> 

<GroundTruth  responsibleGroup=  "A" /> 

<GroundTruthSurpriseProbe  minSurprise= "0 "  maxSurprise= "6" 
increment= "1 "/> 

</ProbeTrial> 

</TrialBlock> 

</ExamPhase> 

Figure  2:  Task  1  <ExamPhase>  element 

The  location  (URL)  of  each  feature  vector  is  contained  in  a  FeatureVectorFile  element;  the  location  of 
both  CSV  and  KML  version  of  the  feature  vector  is  contained  in  the  featureVectorURL_CSV  and 
featureVectorURL_KML  attributes  respectively.  A  URL  may  point  to  a  local  file  (as  shown),  or  to  a 
remote  file  on  a  server. 

"Probes"  in  the  XML  are  intended  to  query  the  participant.  A  ProbeTrial  element  specifies  the  probe 
trial  that  comes  at  the  end  of  a  trial  block;  this  follows  the  presentation  of  each  training  trial. 

ProbeTrial  contains  4  elements:  GroupProbe,  TroopSelectionProbe,  GroundTruth,  and 
GroundTruthSurpriseProbe.  GroupProbe  queries  the  participant  to  provide  the  probability  of  attack  by 
each  group  {A%,  B%}  at  the  location  specified  by  AttackLocation.  The  TroopSelectionProbe  queries  the 
participant  to  make  a  forced  choice  and  select  which  group  to  allocate  troops  against.  The  Groups 
element  provides  the  possible  hypotheses,  i.e.,  the  list  of  groups  (A,  B).  Next,  the  GroundTruth* 
element  specifies  the  group  responsible  for  the  attack,  and  the  GroundTruthSurpriseProbe  element 
asks  the  participant  to  state  their  level  of  "surprise."  Surprise  is  indicated  on  a  scale  defined  by  the 
minSurprise,  moxSurprise,  and  increment  attributes.  In  the  example  above,  the  participant  would 
indicate  surprise  on  a  7-point  integer  scale. 

*Note  that  the  GroundTruth  element  will  not  be  present  during  the  final  Phase  I  evaluation.  Instead, 
ground  truth  information  will  be  automatically  provided  -  at  the  end  of  a  trial  -  by  the  T&E's  Test 
Harness  after  receiving  a  response  from  the  ICArUS  model. 


3.2  Task  2:  Learning  four  groups 

Since  each  task  logically  builds  on  the  last,  only  significant  differences  with  respect  to  the  previous 
task(s)  will  be  highlighted  below  (and  in  future  sections). 

For  convenience,  the  following  task  summary  is  reproduced  here  from  Table  1. 

Task  summary:  Learning  four  groups.  [5  blocks  of  20  trials]  Unlike  Task  1,  there  are  four  different 
groups  (A,  B,  C,  and  D).  Also,  before  reporting  {A%,  B%,  C%,  D%},  the  subject  is  required  to  draw  four 
circles  -  each  representing  the  "two-to-one  boundary"  for  a  group  (i.e.,  attack  by  the  group  is  twice  as 
likely  inside  the  circle  compared  to  outside  the  circle).  After  reporting  the  probabilities  (A%,  B%,  C%, 
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D%)  and  making  a  forced  choice  in  allocating  resources  against  one  group,  the  subject  is  shown  ground 
truth  (A,  B,  C,  or  D)  at  the  attack  location  and  reports  surprise. 

Major  differences:  Each  ProbeTrial  element  contains  a  GroupCirclesProbe  element  (in  addition  to  the 
elements  described  in  Task  1).  The  GroupCirclesProbe  is  intended  to  query  the  participant  to  provide  a 
circle  for  each  group  specified;  each  circle  represents  the  "two-to-one  boundary"  for  the  group  (i.e., 
attack  by  the  group  is  twice  as  likely  inside  the  circle  as  outside  the  circle). 


<!--  Task  2  --> 

<ExamPhase  xsi : type= "ns3 : Task_2_Phase "  examl d= "Sample  Exam" 
name= "  Mission  2"  id =”Task2> 

CTrialBlock  xsi : type= ”ns3 : Task_2_TrialBlock "  trialBlockNum= "1 " 
numPresentationTrials= "1 9 "> 

<FeatureVectorFile  f eatureVectorUrl_CSV= ”task2_l . csv" 
f eatureVectorUrl_KML= "task2_l . kml "/> 

<ProbeTrial  trialNum ="20"> 

<GroupCirclesProbe> 

<Groups>A  B  C  D</Groups> 

</ GroupCirclesProbe> 

<GroupProbe> 

<AttackLocation  locationld ="1"  x="25"  y ="18" 
lat="0 .13025057227020484"  lon=” 0 . 180 9035725 9750 67" /> 
<Groups>A  B  C  D</Groups> 

</ GroupProbe> 

<TroopSelectionProbe> 

<Groups>A  B  C  D</Groups> 

</TroopSelectionProbe> 

<GroundTruth  re sponsible Groups  "B "/ > 

<GroundTruthSurpriseProbe  minSurpris e="0"  maxSurprise= "6" 
increment^ "1 ”/> 

</ProbeTrial> 

</TrialBlock> 

</ExamPhase> 

Figure  3:  Task  2  <ExamPhase>  element 


3.3  Task  3:  Finding  centers 

For  convenience,  the  following  task  summary  is  reproduced  here  from  Table  1. 

Task  summary:  Finding  centers.  [5  blocks  of  20  trials]  Unlike  Task  2,  the  task  is  to  find  group  centers 
along  roads  "as  the  cow  walks"  (rather  than  "as  the  crow  flies").  After  each  block  of  trials,  the  subject 
clicks  four  points  A,  B,  C,  and  D  on  the  roads  -  each  representing  the  "center  of  gravity"  for  a  group 
(i.e.,  maximum  likelihood  location  for  a  group's  center,  given  the  group's  attacks).  The  subject  then 
reports  {A%,  B%,  C%,  D%)  at  the  attack  location.  After  reporting  the  probabilities  and  making  a  forced 
choice  in  allocating  resources,  the  subject  is  shown  ground  truth  (A,  B,  C,  or  D)  at  the  attack  location 
and  reports  surprise. 
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Major  differences:  Task  3  includes  the  addition  of  a  RoadsFile  element  that  specifies  the  URL  location 
of  the  CSV  and  KML  version  of  the  road  network  vector  (see  Section  2.10).  Each  ProbeTrial  element 
contains  a  GroupCentersProbe  element  (in  addition  to  the  elements  described  in  Task  1).  The 
GroupCentersProbe  is  intended  to  query  the  participant  to  provide  the  center  of  gravity  for  each  group 
specified. 


< ! —  Task  3  — > 

<ExamPhase  xsi : type= "ns3 : Task_3_Phase "  examl d= "Sample  Exam" 
nam e= "Mission  3"  id= "Task3"> 

<RoadsFile  f eatureVectorUrl_CSV= "roads . csv” 
f eatureVectorUr 1_KML= "roads . kml "/ > 

CTrialBlock  xsi : type= ”ns3 : Task_3_TrialBlock "  trialBlockNum= ”1 " 
numPresentationTrials= "1 9 "> 

<FeatureVectorFile  f eatureVectorUrl_CSV= ”task3_l . csv” 
f eatureVectorUrl_KML= "task3_l . kml "/> 

<ProbeTrial  trialNum ="20"> 

<GroupCentersProbe> 

<Groups>A  B  C  D</Groups> 

</ GroupCentersProbe> 

<GroupProbe> 

<AttackLocation  locationld ="1"  x=”13"  y =”37" 
lat=" 0 .2677372874443099"  lon="0 . 0940 698577507034 9" /> 
<Groups>A  B  C  D</Groups> 

</ GroupProbe> 

<TroopSelectionProbe> 

<Groups>A  B  C  D</Groups> 

</TroopSelectionProbe> 

<GroundTruth  responsibleGroup= "D" / > 

<GroundTruthSurpriseProbe  minSurpris e="0"  maxSurprise|§|,,6,f 
increment^ "1 "/ > 

</ProbeTrial> 

</TrialBlock> 

</ExamPhase> 

Figure  4:  Task  3  <ExamPhase>  element 


3.4  Task  4:  Scoping  attacks 

For  convenience,  the  following  task  summary  is  reproduced  here  from  Table  1. 

Task  summary:  Scoping  attacks.  [10  trials,  2  stages  per  trial]  Unlike  Task  3,  the  subject  is  given  the 
center  (from  HUMINT)  for  one  group,  along  with  four  possible  locations  1,  2,  3,  and  4  of  attack  by  that 
group.  In  stage  1,  the  subject  is  required  to  estimate  the  probability  of  attack  at  each  location  based  on 
a  normal  distance  decay  function  along  roads.  In  stage  2,  the  subject  is  given  the  regional  boundaries 
for  groups  (SOCINT),  along  with  the  inside/outside  region  attack  likelihoods  from  PROBS.  The  subject  is 
required  to  aggregate  the  prior  (HUMINT)  and  SOCINT  probabilities.  The  subject  allocates  resources  in 
%  across  the  sites,  and  is  shown  ground  truth  (1,  2,  3,  or  4).  The  subject  reports  surprise  after  receiving 
SOCINT,  and  then  again  after  observing  ground  truth. 


14 


Major  differences:  Unlike  Tasks  1  through  3,  which  rely  on  statistical  learning,  Tasks  4  through  6 
require  rule-based  sensemaking.  At  the  beginning  of  a  trial,  the  participant  is  asked  to  estimate  the 
probability  of  attack  at  each  location  via  a  LocationProbe  element.  AttackGroup  specifies  the  group 
being  probed,  and  Locations  specify  candidate  attack  locations,  labeled  by  ID.  The  location  IDs 
specified  in  the  Locations  element  match  the  locations  given  in  the  feature  vector. 

Next,  an  INTLayers  element  contains  the  layers  (one  or  more)  to  be  added,  in  sequence.  In  Task  4,  only 
SOCINT  is  available.  The  INT  data  for  each  attack  location  is  specified  in  the  feature  vector  file  for  the 
trial  (Section  2.9).  SOCINT  at  each  location  is  specified  in  the  feature  vector;  a  "regions"  file  specifies 
SOCINT  at  all  locations.  The  URL  location  of  the  file  is  specified  by  a  RegionsFile  element  (Section  2.11). 
These  data  are  needed  to  visually  render  the  image  and  are  presented  to  human  participants.  After 
the  SOCINT  presentation,  the  participant  is  asked  to  update  the  probability  of  attack  at  each  location 
via  a  LocationProbe  element. 


<!--  Task  4  -- 

<ExamPhase  xs i : type= "ns3 : Task_4_Phase  "  examl d= "Sample  Exam" 
nam e= "Mission  4”  id =  "Task4"> 

CTrial  trialNum= "1 "> 

<FeatureVectorFile  f eatureVectorUrl_CSV= " task4_l . csv" 
f eatureVectorUrl_KML= "task4_l . kml ”  f > 

<RoadsFile  f eatureVectorUrl_CSV= "roads . csv" 
f eatureVectorUrl_KML= "roads . kml "/ > 

<RegionsFile  f eatureVectorUrl_CSV= "SOCINT_l . csv" 
f eatureVectorUrl_KML= "SOCINT_l . kml "/> 

<LocationProbe  > 

<AttackGroup>A</AttackGroup> 

<Locations>l  2  3  4</Locations> 

</LocationProbe> 

<INTLayers> 

<INTLayer> 

<LayerType  xsi : type= "ns3 : SoclntLayer" / > 

<LocationProbe  > 

<AttackGroup>A</AttackGroup> 

<Locations>l  2  3  4</Locations> 

</LocationProbe> 

</INTLayer> 

</INTLayers> 

<TroopAllocationProbe> 

<Locations>l  2  3  4</Locations> 

</TroopAllocationProbe> 

<GroundTruth  attackLocationId= "3 " / > 

<GroundTruthSurpriseProbe  minSurpris e="0"  maxSurprise= "6" 
increment^ "1 "/ > 

</Trial> 

</ExamPhase> 

Figure  5:  Task  4  <ExamPhase>  element 

Finally,  a  TroopAllocationProbe  element  queries  the  participant  to  allocate  troops  across  sites.  A 
GroundTruth  element  specifies  the  actual  attack  location,  and  the 
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GroundTruthSurpriseProbe  element  asks  the  participant/model  to  report  surprise,  as  in  Tasks  1- 

3. 

3.5  Task  5:  Fusing  layers 


For  convenience,  the  following  task  summary  is  reproduced  here  from  Table  1. 

Task  summary:  Fusing  layers.  [10  trials,  4  stages  per  trial]  Unlike  Task  4,  the  subject  is  given  an  attack 
location  (SIGACT)  along  with  group  centers  (HUMINT)  for  four  groups  (A,  B,  C,  D).  The  subject  is  also 
given  the  probabilities  of  attack  by  each  group  {A%,  B%,  C%,  D%},  based  on  the  HUMINT  distance 
function.  The  subject  is  then  given  four  more  INTS  (IMINT,  MOVINT,  SIGINT,  SOCINT),  one  at  a  time, 
along  with  the  likelihoods  (from  PROBS)  for  features  of  INTS.  At  each  stage,  the  subject  updates 
probabilities  {A%,  B%,  C%,  D%}.  After  the  last  stage  the  subject  allocates  resources  in  percentages  {A%, 
B%,  C%,  D%}  against  the  groups.  At  the  end  of  the  trial  the  subject  is  shown  ground  truth  and  reports 
surprise. 

Major  differences:  Task  5  is  similar  to  Task  4,  however,  the  participant  is  asked  to  report  the 
probabilities  of  attack  by  each  group  {A%,  B%,  C%,  D%},  as  opposed  to  the  probability  of  attack  at  each 
location.  Also,  the  subject  does  not  report  surprise  after  each  INT  presentation. 


<!--  Task  5  --> 

<ExamPhase  xsi : type= "ns3 : Task_5_Phase "  examl d= "Sample  Exam" 
nam e= "Mission  5"  id ="Task5"> 

CTrial  trialNum  =”1”> 

<FeatureVectorFile  f eatureVectorUrl_CSV= ”task5_l . csv” 
f eatureVectorUrl_KML= ”task5_l . kml "/> 

<RoadsFile  f eatureVectorUrl_CSV= "roads . csv" 
f eatureVectorUrl_KML= "roads . kml "/ > 

<RegionsFile  f eatureVectorUrl_CSV= ” SOCINT_l . csv" 
f eatureVectorUrl_KML= "SOCINT_l . csv" / > 
<InitialHumintReport> 

<Groups>A  B  C  D</Groups> 

<Probabilities>25  25  25  25</Probabilities> 
</InitialHumintReport> 

<INTLayers> 

<INTLayer> 

<LayerType  xsi : type= "ns3 : ImintLayer" / > 
<GroupProbe> 

<AttackLocation  locationld ="1 "/> 

<Groups>A  B  C  D</Groups> 

</ GroupProbe> 

</INTLayer> 

<INTLayer> 

<LayerType  xsi : type= "ns3 : MovintLayer" / > 
<GroupProbe> 

<AttackLocation  locationld ="1 "/> 

<Groups>A  B  C  D</Groups> 

</ GroupProbe> 

</INTLayer> 
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<INTLayer> 

<LayerType  xsi : type= ”ns3 : SigintLayer"> 

<Group>B</Group> 

</LayerType> 

<GroupProbe> 

<AttackLocation  locationId= "1 ”/> 

<Groups>A  B  C  D</Groups> 

</ GroupProbe> 

</INTLayer> 

<INTLayer> 

<LayerType  xsi : type=  ”ns3 : SocintLayer" / > 

<GroupProbe> 

<AttackLocation  locationId= "1 "/> 

<Groups>A  B  C  D</Groups> 

</ GroupProbe> 

</INTLayer> 

</INTLayers> 

<TroopAllocationProbe> 

<Groups>A  B  C  D</Groups> 

</TroopAllocationProbe> 

<GroundTruth  responsibleGroup= ”C" /> 

<GroundTruthSurpriseProbe  minSurprise=  "0  "  maxSurprisel|,,6,f 
increment^ "1  "/ > 

</Trial> 

</ExamPhase> 

Figure  6:  Task  5  <ExamPhase>  element 

Another  key  difference  is  that  layers  are  presented  sequentially  (IMINT,  MOVINT,  SIGINT,  and  SOCINT, 
see  Figure  6).  All  INT  data  is  contained  in  the  feature  vector  (see  Section  2.9).  For  SIGINT, 
theparticipant  must  specify  which  group  s/he  would  like  to  collect  SIGINT  on;  only  1  group  can  be 
chosen  per  trial. 

After  each  layer  is  presented,  the  participant  will  be  asked  to  update  their  probability  estimates  and 
report  surprise.  As  in  Task  4,  after  the  last  layer  is  presented,  the  participant  will  allocate  troops,  see 
the  ground  truth  (responsible  group),  and  report  surprise. 


3.6  Task  6:  Choosing  layers 

For  convenience,  the  following  task  summary  is  reproduced  here  from  Table  1. 

Task  summary:  Choosing  layers.  [10  trials,  3  stages  per  trial]  Task  6  is  much  like  Task  5,  except  the 
subject  decides  which  layer  of  INT  (IMINT,  MOVINT,  SIGINT,  or  SOCINT)  to  receive  on  each  of  three 
stages,  after  receiving  the  initial  HUMINT.  The  criterion  for  each  choice  is  to  maximize  the  expected 
amount  of  useful  information  to  be  gained  from  the  choice  of  an  INT. 
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Major  differences:  Task  6  is  almost  identical  to  Task  5,  except  that  layers  are  chosen  by  the 
model/participant  (user  choice)  instead  of  being  presented  sequentially.  The  total  number  of  layers  to 
select  is  specified  in  the  numLayersToShow  attribute  within  each  Trial  element. 


< ! —  Task  6  --> 

<ExamPhase  xs i : type= "ns  3 : Task_  6_Phase "  examName= "Sample  Exam” 
name= "Mission  6"  id=”Task6"> 

<Trial  trialNum= ”1 ”  numLayersToShow =”1”> 

<FeatureVectorFile  f eatureVectorUrl_CSV= ”task6_l . csv” 
f eatureVectorUrl_KML= ” task6_l . kml ”  / > 

<RoadsFile  f eatureVectorUrl_CSV= "roads . csv" 
f eatureVectorUrl_KML= "roads . kml "/ > 

<RegionsFile  f eatureVectorUrl_CSV= ” S0CINT_1 . csv" 
f eatureVectorUrl_KML= ” S0CINT_1 . csv" /> 
<InitialHumintReport> 

<Groups>A  B  C  D</Groups> 

<Probabilities>25  25  25  25</Probabilities> 
</InitialHumintReport> 

<INTLayers> 

<INTLayer> 

<LayerType  xsi : type= "ns3 : ImintLayer" / > 
<GroupProbe> 

<AttackLocation  locationId= "3 "/ > 

<Groups>A  B  C  D</Groups> 

</ GroupProbe> 

</INTLayer> 

<INTLayer> 

<LayerType  xsi : type= ”ns3 : MovintLayer" /> 
<GroupProbe> 

<AttackLocation  locationId=  "3 "/ > 

<Groups>A  B  C  D</Groups> 

</ GroupProbe> 

</INTLayer> 

<INTLayer> 

<LayerType  xsi : type= "ns3 : SigintLayer"> 
<Group>A</ Group> 

</LayerType> 

<GroupProbe> 

<AttackLocation  locationId=  "3 " / > 

<Groups>A  B  C  D</Groups> 

</ GroupProbe> 

</INTLayer> 

<INTLayer> 

<LayerType  xsi : type= "ns3 : SigintLayer"> 
<Group>B</ Group> 

</LayerType> 

<GroupProbe> 

<AttackLocation  locationId=  "3 " / > 

<Groups>A  B  C  D</Groups> 

</ GroupProbe> 

</INTLayer> 

<INTLayer> 

<LayerType  xsi : type= "ns3 : SigintLayer"> 
<Group>C</Group> 
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</LayerType> 

<GroupProbe> 

<AttackLocation  locationId= "3 "/ > 

<Groups>A  B  C  D</Groups> 

</ GroupProbe> 

</INTLayer> 

<INTLayer> 

<LayerType  xsi : type= "ns3 : SigintLayer"> 

<Group>D</ Group> 

</LayerType> 

<GroupProbe> 

<AttackLocation  locationId=  "3 "/ > 

<Groups>A  B  C  D</Groups> 

</ GroupProbe> 

</INTLayer> 

<INTLayer> 

<LayerType  xsi : type= ”ns3 : SocintLayer" /> 
<GroupProbe> 

<AttackLocation  locationId=  "3 "/ > 

<Groups>A  B  C  D</Groups> 

</ GroupProbe> 

</INTLayer> 

</INTLayers> 

<TroopAllocationProbe> 

<Groups>A  B  C  D</Groups> 

</TroopAllocationProbe> 

<GroundTruth  responsibleGroup= "D"/ > 

<GroundTruthSurpriseProbe  minSurpris e="0"  maxSurprise- "6” 
increment^ "1 "/ > 

</Trial> 

</ExamPhase> 

</IcarusEvaluation_CPDl> 

Figure  7:  Task  6  <ExamPhase>  element 


3.7  Task  7:  Self-initiated  sensemaking 

For  convenience,  the  following  task  summary  is  reproduced  here  from  Table  1. 

Task  summary:  Self-initiated  sensemaking.  [15  trials]  Task  7  is  a  combination,  in  part,  of  Tasks  4  and  6, 
but  with  a  focus  on  self-initiated  sensemaking.  Participants  will  be  given  the  centers  of  four  groups  (A- 
D)  and  four  possible  attack  sites  (1-4).  As  in  Task  4,  the  primary  task  is  to  predict  the  location  of  attack 
and  to  allocate  resources  to  each  location  {@1%,  @2%,  @3%,  @4%}.  But  unlike  Task  4,  participants 
will  decide  if  and  when  to  seek  additional  data,  as  well  as  which  data  (INT)  to  seek.  See  the  Phase  1 
Challenge  Problem  Design  and  Test  Specification  for  a  more  detailed  description  of  Task  7,  including  a 
summary  of  the  major  structural  differences  between  Task  7  and  Tasks  1-6. 

Major  differences:  Task  7  departs  from  Tasks  4  and  6  in  several  ways.  For  one,  the  group  center 
locations,  road  network,  and  SOCINT  regions  are  fixed  for  the  entire  task.  Thus,  the  RoadsFile  and 
RegionsFile  elements  at  the  ExamPhase  element  level  define  the  feature  vector  files  for  the  road 
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network  and  SOCINT  regions  for  all  trials  in  the  task. 


The  InitialResponsibleGroup  element  defines  the  group  responsible  for  the  last  attack  wave.  This  group 
is  always  shown  on  the  first  trial.  Initial  group  attack  probabilities  are  provided  to  the  participant  at 
the  start  of  the  task,  and  it  is  up  to  the  participant  whether  to  update  them  in  subsequent  trials;  the 
participant  may  not  update  them  on  the  first  trial.  The  InitialCredits  element  next  defines  the  starting 
credits  allotted  to  the  participant,  and  the  CorrectPredictionCredits  element  defines  the  maximum 
number  of  credits  awarded  for  correctly  predicting  the  attack  location.  The  actual  number  of  credits 
awarded  is  computed  as  the  troop  allocation  percent  at  the  actual  attack  location  multiplied  by 
CorrectPredictionCredits. 

Each  trial  first  contains  a  FeatureVectorFile  element  that  specifies  the  location  of  the  feature  vector  file 
defining  the  attack  locations  and  group  center  locations  (the  group  center  locations  will  remain  fixed 
for  the  duration  of  the  task,  but  SIGINT  information  on  each  group  may  be  different  for  each  trial).  The 
GroupProbe  element  next  queries  the  participant  to  optionally  update  the  probability  of  attack  by  each 
group  (before  the  attack  locations  are  shown).  While  a  GroupProbe  element  is  present  on  the  first 
trial,  the  participant  will  simply  be  shown  the  initial  group  attack  probabilities  and  will  not  have  an 
opportunity  to  update  them.  Probabilities  are  initialized  based  on  the  initial  responsible  group;  for 
example,  if  the  initial  responsible  group  is  A,  the  group  attack  probabilities  would  be  {85%,  5%,  5%, 

5%). 


The  attack  locations  are  then  presented  and  the  LocationProbe  element  queries  the  participant  to 
provide  the  probability  of  attack  at  each  location.  Next,  a  TroopAllocationProbe  element  queries  the 
participant  to  allocate  troops  at  each  location. 

The  ResponsibleGroupShown  element  indicates  whether  the  group  responsible  for  the  attack  will  be 
shown  (the  actual  attack  location  is  always  shown).  In  trial  1,  the  responsible  group  will  always  be 
presented  at  the  start  of  the  trial.  Both  the  responsible  group  and  attack  location  are  specified  in  the 
GroundTruth  element.  Finally,  the  INTLayers  element  defines  the  INT  layers  that  may  be  purchased 
provided  the  participant  has  enough  credits  (the  cost  of  each  layer  is  specified  in  the  costCredits 
attribute).  The  participant  may  either  purchase  one  or  more  layers  to  see  or  continue  to  the  next  trial. 
The  participant  will  not  be  given  the  option  to  purchase  INT  layers  on  the  last  trial. 


<!--  Task  7  --> 

<ExamPhase  xs i : type= "ns3 : Task_  7_Phase "  examld= "Sample  Exam" 
nam e= "Mission  7"  id= "Task 7"> 

<FeatureVec tor File  f eatureVectorUrl_CSV= "task 7_phase . csv” 
f eatureVectorUrl_KML= "task 7_phase . kml "/> 

<RoadsFile  f eatureVectorUrl_CSV= "roads . csv" 
f eatureVectorUrl_KML= "roads . kml "/ > 

<RegionsFile  f eatureVectorUrl_CSV= "SOCINT_l . csv" 

f eatureVectorUrl_KML= ” SOCINT_l . csv" /> 
<InitialResponsibleGroup>A</InitialResponsibleGroup> 
<InitialCredits>10</InitialCredits> 

<CorrectPredictionCredits>l</ CorrectPredictionCredits> 
<Trial  trialNum  =”1”> 

<FeatureVectorFile  f eatureVectorUrl  CS V="task7  l.csv” 


20 


f eatureVectorUr 1_KML= "task 7_1 . kml "/ > 

<GroupProbe> 

<Groups>A  B  C  D</Groups> 

</ GroupProbe> 

<LocationProbe> 

<Locations>l  2  3  4</Locations> 

</LocationProbe> 

<TroopAllocationProbe> 

<Locations>l  2  3  4</Locations> 
</TroopAllocationProbe> 

<ResponsibleGroupShown>true</ResponsibleGroupShown> 
<GroundTruth  responsibleGroup= "A "  attackLocationld ="4"/> 
<INTLayers> 

<INTLayer  costCredits ="1"> 

<LayerType  xsi : type= "ns3 : ImintLayer" / > 
</INTLayer> 

<INTLayer  costCredits ="1"> 

<LayerType  xsi : type= "ns3 : MovintLayer" / > 
</INTLayer> 

<INTLayer  costCredits= "2 "> 

<LayerType  xsi : type= ”ns3 : SigintLayer"> 
<Group>A</Group> 

</LayerType> 

</INTLayer> 

<INTLayer  costCredits^ "2 "> 

<LayerType  xsi : type= "ns3 : SigintLayer"> 
<Group>B</Group> 

</LayerType> 

</INTLayer> 

<INTLayer  costCredits^ "2 "> 

<LayerType  xsi : type= ”ns3 : SigintLayer"> 
<Group>C</Group> 

</LayerType> 

</INTLayer> 

<INTLayer  costCredits= "2 "> 

<LayerType  xsi : type= "ns3 : SigintLayer"> 
<Group>D</Group> 

</LayerType> 

</INTLayer> 

ciNTLayer  costCredits^ "2 "> 

<LayerType  xsi : type=  ”ns3 : SocintLayer" / > 
</INTLayer> 

</INTLayers> 

</Trial> 

</ExamPhase> 

Figure  8:  Task  7  <ExamPhase>  element 


3.8  Feature  Vector  Format:  Tasks  1  -3 

In  Tasks  1-3,  attacks  will  be  defined  as  Placemarks  in  a  KML  document  as  shown  in  Figure  8  below 
(see  http://code.gooRle.com/apis/kml/documentation/kmlreference.html  for  more  information  on 
KML).  The  KML  document  will  contain  an  ordered  series  of  one  or  more  Placemark  elements 
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corresponding  to  each  attack  presentation.  Each  Placemark  element  will  contain  Data  elements  in  the 
ExtendedData  section  that  specify  the  Objectld  and  ObjectType  values.  In  the  case  of  Tasks  1-3,  the 
ObjectID  will  indicate  the  group  responsible  for  the  attack,  and  the  ObjectType  will  always  be 
"Location".  The  Point  element  will  contain  the  longitude  and  latitude  of  the  attack  location  in  degrees. 
Longitudes  and  latitudes  may  be  converted  to  grid  coordinates  using  the  formulas: 

nr(longitude  —  bottomLeftLon )  nr(latitude  —  bottomLeftLat ) 

x  =  - ,  y  =  - 

180  *  milesPerGridUnit  180  *  milesPerGridUnit 

where  r  is  the  mean  radius  of  the  earth  in  miles  (app.  3,959  miles),  and  bottomLeftLat,  bottomLeftLon, 
and  milesPerGridUnit  are  specified  in  the  exam's  GridSize  element. 

All  distances  supplied  by  an  ICArUS  model  should  be  provided  in  grid  units. 


<?xml  version ="1.1"  encoding= "UTF-8 "  standalone= "yes ”?> 

<kml  xmlns= "http : //www. opengis . net/kml/2 . 2 " 

xmlns : gx= "http : //www . google . com/ kml / ext / 2 . 2" 

xmlns : atom= "http : / /www . w3 . org/ 2005/ Atom  " 

xmlns : xal= "urn : oasis : names : tc : ciq : xsdschema : xAL : 2 . 0 "> 

<Document> 

<name>Task  l</name> 

<Placemark> 

<ExtendedData> 

<Data  name= "ObjectId"> 

<value>A</value> 

</Data> 

<Data  name= "Object Type "> 

<value>Location</value> 

</Data> 

</ExtendedData> 

<Point> 

<coordinates>0 . 09551708633148355, 0. 13603948659332504c/ coordinate s> 
</Point> 

</Placemark> 

<Placemark> 

<ExtendedData> 

<Data  name= "Obj ect Id ”> 

<value>B</value> 

</Data> 

<Data  name= "Obj ect Type "> 

<value>Location</value> 

</Data> 

</ExtendedData> 

<Point> 

<coordinates>0 .1157782864624043, 0.083939257  6852  4  312</coordinates> 
</Point> 

</Placemark> 

</Document> 

</kml> 

Figure  9:  Sample  feature  vector  for  Tasks  1-3  (KML  version) 
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While  developers  are  encouraged  to  adopt  the  KML  feature  vector  format,  a  CSV  (comma-separated 
value)  format  will  also  be  provided.  Each  CSV  feature  vector  contains  4  columns  as  defined  below. 
The  example  in  Figure  10  below  contains  9  attack  presentations. 


Figure  10:  Sample  feature  vector  for  Tasks  1-3  (CSV  version) 

Name:  ObjectID 

Description:  Object  identification;  in  this  case  a  group. 

Format:  String 
Range:  [A,  B,  C,  D] 

Name:  ObjectType 

Description:  The  type  of  object  being  identified. 

Format:  String 
Range:  [Location] 

Name:  X 

Description:  The  X-coordinate  of  the  data. 

Format:  Scalar 
Range:  [0  ...  GridWidth] 

Name:  Y 

Description:  The  Y-coordinate  of  the  data. 

Format:  Scalar 
Range:  [0  ...  GridHeight] 

Note:  Refer  to  Appendix  B  for  guidance  to  obtain  example  feature  vectors. 

3.9  Feature  Vector  Format:  Tasks  4-7 

The  location  data  for  Tasks  4-7  will  also  be  defined  as  Placemarks  in  a  KML  document  as  shown  in 
Figure  11  below.  The  KML  document  will  contain  a  list  of  Placemark  elements  corresponding  to  either 
group  center  locations  or  attack  locations.  The  Placemark  element  will  always  contain  at  least  two 
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Data  elements  in  the  ExtendedData  section  that  specify  the  Objectld  and  ObjectType  values.  The 
ObjectID  will  provide  either  the  group  information  or  a  location  ID,  and  the  object  type  will  either  be 
"Group"  or  "Location"  respectively.  There  may  be  additional  Data  elements  that  specify  the  various 
INT  layers  at  the  group  center  or  attack  location.  Such  Data  elements  include: 

Name:  "I  Ml  NT" 

Values:  [Government,  Military] 

Name:  "MOVINT" 

Values:  [SparseTraffic,  DenseTraffic] 

Name:  "SIGINT" 

Values:  [Silent,  Chatter] 

Name:  "SOCINT" 

Values:  [A,  B,  C,  D] 

The  Point  element  will  contain  the  longitude  and  latitude  of  the  group  center  or  attack  location  in 
degrees  (see  Section  2.8  for  information  on  translating  longitudes  and  latitudes  to  grid  coordinates). 


<?xml  version =”1.1"  encoding ="UTF-8"  standalone^ "yes "?> 

<kml  xmlns= "http : //www. opengis . net/kml/2 . 2 " 

xmlns : gx= "http : //www . google . com/ kml / ext / 2 . 2" 

xmlns : atom= "http : //www. w3 . org/ 2005/ Atom  " 

xmlns : xal= "urn : oasis : names : tc : ciq: xsdschema : xAL : 2 .  0 "> 

<Document> 

<name>Task  5</name> 

<Placemark> 

<ExtendedData> 

<Data  name= "ObjectId"> 

<value>l</value> 

</Data> 

<Data  name= "Object Type "> 

<value>Location</value> 

</Data> 

<Data  name= " IMINT"> 

<value>Military</value> 

</Data> 

<Data  name= "MOVINT "> 

<value>SparseTraf f ic</value> 

</Data> 

<Data  nam e="SOCINT"> 

<value>A</value> 

</Data> 

</ExtendedData> 

<Point> 

<coordinates>0 . 5716552894081212 , 0.5427107177 92  52  01</coordinates> 
</Point> 

</Placemark> 

<Placemark> 

<ExtendedData> 
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<Data  name= "Obj ect Id "> 

<value>A</value> 

</Data> 

<Data  name= "Obj ect Type "> 

<value>Group</value> 

</Data> 

<Data  nam e="SIGINT"> 

<value>Silent</value> 

</Data> 

</ExtendedData> 

<Point> 

<coordinates>0 .4196962884262156, 0. 5788914323120214</coordinates> 
</Point> 

</Placemark> 

<Placemark> 

<ExtendedData> 

<Data  nam e="0bjectld"> 

<va lue >B< / value > 

</Data> 

<Data  name="Obj  ectType"> 

<value>Group</value> 

</Data> 

<Data  nam e="SIGINT"> 

<value>Chatter</value> 

</Data> 

</ExtendedData> 

<Point> 

<coordinates>0 .10130600065460375, 0. 665725147158824 6</ coordinate s> 
</Point> 

</Placemark> 

<Placemark> 

<ExtendedData> 

<Data  nam e= "Obj ectld"> 

<value>C</value> 

</Data> 

<Data  nam e="Obj ectType"> 

<value>Group</value> 

</Data> 

<Data  nam e="SIGINT"> 

<value>Silent</value> 

</Data> 

</ExtendedData> 

<Point> 

<coordinates>0 .37  627  943100281397,0. 13025057227  020484c/ coordinate s> 
</Point> 

</Placemark> 

<Placemark> 

<ExtendedData> 

<Data  name= ”Obj ectld"> 

<value>D</value> 

</Data> 

<Data  name=" Obj ectType"> 

<value>Group</value> 

</Data> 

<Data  ndLme="SIGINT"> 

<value>Silent</value> 
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</Data> 

</ExtendedData> 

<Point> 

<coordinates>0 . 16643128678970615, 0.19537585840530722</ coordinate s> 
</Point> 

</Placemark> 

</Document> 

:/kml> 

Figure  11:  Sample  feature  vector  for  Tasks  4-7  (KML  version) 

While  developers  are  encouraged  to  adopt  the  KML  feature  vector  format,  a  CSV  (comma-separated 
value)  format  will  also  be  provided.  Each  CSV  feature  vector  contains  8  columns  as  defined  below. 
The  example  in  Figure  11  below  is  for  Task  5. 


ObjectID 

ObjectType 

X 

Y 

SOCINT 

IMINT 

MOVINT 

SIGINT 

1 

Location 

79 

75 

A 

1 

1 

[] 

A 

Group 

58 

80 

A 

[] 

[] 

2 

B 

Group 

14 

92 

B 

[] 

[] 

1 

C 

Group 

52 

18 

C 

[] 

[] 

1 

D 

Group 

23 

27 

D 

[] 

[] 

1 

Figure  12:  Sample  feature  vector  for  Tasks  4-7  (CSV  version) 


Name:  ObjectID 

Description:  Object  identification 
Format:  String 

Range:  [A,  B,  C,  D]  for  groups,  and  [1,  2,  3,  4]  for  locations. 
Name:  ObjectType 

Description:  The  type  of  object  being  identified. 

Format:  String 
Range:  [Location,  Group] 

Name:  X 

Description:  The  X-coordinate  of  the  data. 

Format:  Scalar 
Range:  [0  ...  GridWidth] 

Name:  Y 

Description:  The  Y-coordinate  of  the  data. 

Format:  Scalar 
Range:  [0  ...  GridHeight] 


Name:  SOCINT 

Description:  Sociocultural  intelligence. 
Format:  String 
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Range:  [A,  B,  C,  D],  "A"  refers  to  Group  A's  'territory',  "B"  refers  to  Group  B's  territory  etc. 
Name:  IMINT 

Description:  Image  intelligence;  facility  identity. 

Format:  Scalar 

Range:  [1,  2],  where  "1"  denotes  a  government  facility  and  "2"  denotes  a  military  facility. 
Name:  MOVINT 

Description:  Movement  intelligence;  dense  vs.  sparse  traffic. 

Format:  Scalar 

Range:  [1,  2],  where  "1"  denotes  sparse  traffic  and  "2"  denotes  dense  traffic. 

Name:  SIGINT 

Description:  Signals  Intelligence. 

Format:  Scalar 

Range:  [1,  2],  where  "1"  denotes  silence  (no  intelligence),  and  "2"  denotes  chatter  (significant 
intelligence) 


3.10  Feature  Vector  Format:  Roads 


Roads  will  also  be  defined  as  Placemarks  in  a  KML  document  as  shown  in  Figure  13  below.  The  KML 
document  will  contain  a  list  of  Placemark  elements  for  each  road.  The  Placemark  element  will  always 
contain  two  Data  elements  in  the  ExtendedData  section;  one  that  specifies  the  Objectld  and  another 
that  specifies  the  ObjectType.  The  ObjectID  provides  the  ID  for  the  road;  the  object  type  will  be 
"Road". 

The  LineString  element  will  contain  an  ordered  list  of  longitudes  and  latitudes  that  define  the  points 
along  the  road  (see  Section  2.8  for  information  on  translating  longitudes  and  latitudes  to  grid 
coordinates). 


<?xml  version ="1.1"  encoding= "UTF-8 "  standalone= "yes "?> 
<kml  xmlns= "http : //www. opengis . net/kml/2 . 2 " 
xmlns : gx= "http :/ /www . google . com/ kml / ext / 2 . 2" 
xmlns : atom= "http : //www . w3 . org/2005/ Atom  ” 
xmlns : xal= "urn : oasis : names : tc : ciq: xsdschema : xAL : 2 . 0 "> 
<Document> 

<name>Roads</name> 

<Placemark> 

<ExtendedData> 

<Data  name= "Obj ect Id ”> 

<value>l</value> 
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</Data> 

<Data  name= rr  Ob  j  ectType”> 

<value>Road</value> 

</Data> 

</ExtendedData> 

<LineString> 

<coordinates>0 .3618071451950134,0.04341685742340161 
0.3618071451950134,0 . 05065300032730188 
0.3618071451950134,0.05788914323120215 
0.3618071451950134,0.06512528613510242 
0.3618071451950134,0.07236142903900268 

</ coordinates> 

</LineString> 

</Placemark> 

</Document> 

</kml> 

Figure  13:  Sample  feature  vector  for  roads  (KML  version) 

While  developers  are  encouraged  to  adopt  the  KML  feature  vector  format,  a  CSV  (comma-separated 
value)  format  will  also  be  provided  as  shown  in  Figure  14  below. 


ObjectID 

ObjectType 

X 

Y 

1 

Road 

50 

6 

1 

Road 

52 

5 

1 

Road 

63 

34 

1 

Road 

67 

40 

1 

Road 

72 

48 

1 

Road 

74 

53 

1 

Road 

79 

62 

1 

Road 

79 

82 

1 

Road 

79 

91 

Figure  14:  Sample  feature  vector  for  roads  (CSV  version) 
Name:  ObjectID 

Description:  Object  identification 
Format:  Integer 
Range:  [0  ...  100] 

Name:  ObjectType 

Description:  The  type  of  object  being  identified. 

Format:  String 
Range:  [Road] 

Name:  X 

Description:  The  X-coordinate  of  the  data. 

Format:  Scalar 
Range:  [0  ...  GridWidth] 
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Name:  Y 

Description:  The  Y-coordinate  of  the  data. 

Format:  Scalar 

Range:  [0  ...  GridHeight] 


3.11  Feature  Vector  Format:  Regions 


Note  on  regions:  While  models  are  not  expected  to  read  region  files,  they  will  be  presented  visually  to 
human  participants.  A  description  of  the  format  is  provided  here  for  completeness. 

Regions  will  also  be  defined  as  Placemarks  in  a  KML  document  as  shown  in  Figure  15  below.  The  KML 
document  will  contain  a  list  of  Placemark  elements  that  define  a  series  of  polygons  for  all  SOCINT 
regions  in  the  grid.  The  Placemark  element  will  always  contain  one  Data  element  in  the  ExtendedData 
section  that  specifies  the  group  the  SOCINT  region  is  for.  Multiple  polygons  may  be  defined  for  a  single 
group. 

The  Polygon  element  will  contain  vertices  defining  a  polygon  that  encompasses  the  region  (see  Section 
2.8  for  information  on  translating  longitudes  and  latitudes  to  grid  coordinates). 


<?xml  version=" 1 . 1 "  encoding="UTF-8 "  standalone="yes " ?> 

<kml  xmlns= "http : //www. opengis . net/kml/2 . 2 " 

xmlns : gx= "http : //www . google . com/ kml / ext / 2 . 2" 

xmlns : atom= "http : //www. w3 . org/ 2005/ Atom  " 

xmlns : xal= "urn : oasis : names : tc : ciq : xsdschema : xAL : 2 . 0 ”> 

<Document> 

<Document> 

<name>Regions</name> 

<Placemark> 

<styleUrl>#3pt</styleUrl> 

<ExtendedData> 

<Data  name= "SOCINT"> 

<value>C</value> 

</Data> 

</ExtendedData> 

<Polygon> 

<outerBoundaryIs> 

<LinearRing> 

<tessellate>l</ tessellate> 

<coordinates>0 .24313440157104899, 0.004341685742340161 
0 .24747608731338916, 0 . 004341685742340161 
0.24747608731338916, 0. 008683371484680322 
0.24313440157104899, 0. 008683371484680322 
0.24313440157104899, 0.004341685742340161 
</ coordinates> 

</LinearRing> 

</ outerBoundaryIs> 

</Polygon> 

</Placemark> 

</Document> 

</Document> 
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:/kml> 

Figure  15:  Sample  feature  vector  for  regions  (KML  version) 

A  CSV  (comma-separated  value)  format  is  also  provided  as  shown  in  Figure  16  below.  The  SOCINT 
value  (group)  of  each  grid  coordinate  (X,  Y)  is  specified.  Note  that  SOCINT  values  may  be  specified  as 
either  numbers  (1,  2,  3,  4)  that  correspond  to  groups  (A,  B,  C,  D),  or  as  the  groups  themselves  as  shown 
in  Figure  16. 
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Figure  16:  Sample  feature  vector  for  regions  (CSV  version) 
Name:  X 

Description:  The  X-coordinate  of  the  data. 

Format:  Scalar 
Range:  [0  ...  GridWidth] 

Name:  Y 

Description:  The  Y-coordinate  of  the  data. 

Format:  Scalar 
Range:  [0  ...  GridHeight] 

Name:  SOCINT 
Description:  The  group 
Format:  String  or  Scalar 
Range:  [A,  B,  C,  D]  or  [1,2, 3, 4] 
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4  Output  Format 


A  participant's  or  model's  responses  to  an  exam  will  be  recorded  in  the  same  XML  document  that 
defines  the  exam.  A  ResponseGenerator  element  at  the  beginning  of  the  document  provides 
information  about  the  participant/ICArUS  model  that  generated  the  responses;  a  unique  Siteld  will  be 
provided  to  each  team,  and  each  team  may  use  the  ResponseGeneratorld  element  to  specify  the 
model  version  ID  (for  example,  there  may  be  multiple  IDs  corresponding  to  different  versions  of  a 
model  running  with  different  parameters).  It  is  preferable  to  avoid  using  an  underscore  character  ('_') 
in  the  site  ID  or  response  generator  ID  names  as  the  underscore  is  used  as  a  field  delineator. 

The  document  may  also  contain  StartTime  and  EndTime  elements  that  indicate  when  the 
participant/model  started  and  finished  the  exam.  The  ExamPhase  elements  for  each  task  may  also 
contain  StartTime  and  EndTime  elements  that  indicate  when  the  participant/model  started  and 
finished  the  task.  Responses  to  the  trials  in  each  task  phase  are  contained  within  the  phase  and  are 
described  in  Sections  4.1  -  4.6  below.  Appendix  C  contains  an  entire  sample  output  document.  While 
the  example  given  in  Appendix  C  shows  responses  for  all  trials  in  all  tasks  in  the  exam,  a  model  should 
only  send  an  exam  document  containing  results  for  a  single  task  at  a  time. 


<?xml  version ="1.1"  encoding= "UTF-8 "  standalone= "yes "?> 

<ns3 : IcarusEvaluation_CPDl  xmlns : ns2= " IcarusCPD_Base "  xmlns : ns3= " IcarusCPD_l " 
examTimeStamp= "2011-07-27T12:50: 45.571-04:00"  name= "Sample  Exam " 
id =" Sample  Exam" 

startTim e=" 2011-07 -27T13: 50 : 45 .571-04:00" 
endTime= "2011-07-27T1 6:50:45. 571-04:00 "> 

<!--  ICArUS  System  information  --> 

<ResponseGenerator> 

<HumanSub j  ect>f alse</HumanSub j  ect> 

<Respons eGene ra tor Id>Model2< /Response Genera tor Id> 

<SiteId>Teaml</ Siteld> 

</ResponseGenerator> 

<!--  Task  1  Response  --> 

<ExamPhase  xsiityp e="ns3 : Task_l_Phase"  examld =" Sample  Exam" 
name= "Mission  1"  id= ”Task3" 
startTim e=" 2011-07 -27T13 : 50 : 45 . 571-04 : 00" 
endTime= "201 1-07 -27T1 4:20:45. 571-04:00  ”> 

<ExamPhase> 


</ns3 : IcarusEvaluation_CPDl> 

Figure  17:  Overview  of  an  exam  document  with  ICArUS  model  responses 


Note  on  timing  data:  The  trialTime_ms  attribute  and  trialPartTime_ms  attributes  capture  the  amount 
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of  time  taken  by  a  participant  on  the  overall  trial  and  each  part  of  the  trial.  The  time_ms  attributes  in 
GroupAttackProbability  and  TroopAllocation  elements  capture  the  amount  of  time  spent  adjusting 
input  controls  when  providing  probability  judgments  and  troop  allocations.  The  layerSelectionTime_ms 
attribute  in  the  INTLayerShown  element  captures  the  amount  of  time  spent  selecting  an  INT  layer  in 
Task  6.  While  this  information  is  captured  for  human  participants,  ICArUS  models  are  not  required  to 
report  this  information. 

Note  on  normative  response  data  and  SI  and  SI  scores:  The  normative  responses  for  each  stage  of 
each  trial  and  the  scores  for  assessing  probabilities  (SI)  and  allocating  troops  (S2)  are  also  calculated 
and  embedded  in  the  response  elements.  The  ResponseFeedback  element  for  a  trial  contains  the 
overall  SI  and  S2  scores  for  the  trial.  In  this  case  of  trials  with  multiple  probability  assessments  (such  as 
those  in  Tasks  4-6),  the  overall  SI  score  is  the  average  of  each  individual  SI  score  for  each  probability 
assessment.  ICArUS  models  are  not  expected  to  report  this  information.  The  example  normative 
solutions  in  this  document  are  used  only  to  illustrate  XML  formats  and  are  not  actual  normative 
solutions.  Please  consult  the  appendix  "Normative  Solutions"  in  the  "Sample  Human  Behavioral  Data" 
document  for  details  on  how  normative  solutions  are  calcualted  and  consult  the  "Phase  1  Challenge 
Problem  Design  and  Test  Specification"  document  for  details  on  how  scores  are  calculated. 


4.1  Task  1  Output 

The  TrialResponse  element,  contained  within  the  ProbeTrial  element  (see  Figure  18),  contains  the 
participant's  response  to  each  probe  trial.  The  TrialResponse  element  next  includes  a  GroupResponse 
element,  which  contains  the  probability  that  each  group  is  responsible  for  the  attack.  The 
TroopSelectionProbeResponse  element  next  indicates  the  group  that  the  participant  chose  to  allocate 
troops  against,  and  the  GroundTruthSurpriseResponse  element  contains  the  participant's  "surprise" 
after  being  shown  ground  truth. 

Normative  Response  Data  and  Scores:  The  ResponsFeedback  element  contains  the  participant's  overall 
SI  and  S2  scores  for  the  trial.  The  NormativeProbsCumulative  element  in  the  GroupResponse  element 
contains  the  normative  probabilities,  and  the  ProbabilitiesScore_sl  element  contains  the  participant's 
SI  score  for  the  probability  assessment.  The  TroopAllocationScore_s2  element  contains  the 
participant's  S2  score  for  the  troop  allocation.  The  AttackDispersionParameters  element  contains  the 
"dispersion"  parameters  for  each  group  (e.g.,  base  rates,  mean  locations)  based  on  the  distribution  of 
attacks  that  have  been  presented. 


<!--  Task  1  Response  --> 

<ExamPhase  xsi : type= "ns3 : Task_l_Phase "  examld= "Sample  Exam" 
nam e=”M±ss±on  1"  id ="Taskl"  startTime=”2011-07-27T13 : 50 : 45 . 571-04  :  00” 
endTime= ”201 1-07-27T1 4:20:45. 571-04:00 ”> 

<TrialBlock  xsi : type= ”ns3 : Task_l_TrlalBlock ”  trialBlockNum= ”1 ” 
numPresentationTrials= ”9 ”> 

<FeatureVectorFile  f eatureVectorUrl_CSV= ” taskl_l . csv ” 
f eatureVectorUrl_KML= ”taskl_l . kml ”/ > 

<ProbeTrial  trialNum =”10”> 
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<JttackDispersionParameters> 

<Parameters  baseRate=  "0 . 5  "  group="A"  sigmaX=  ”5 .  0  " 
sigmaY= "5 . 0 "  theta ="0.0"> 

<CenterLocation  locationId= "A "  x="25.0"  y ="75.0"  /> 
</Parameters> 

<Parameters  baseRate=  "0 . 5  "  group  =T,B"  sigmaX=  "2 . 5  " 
sigmaY= ”2 . 5 "  theta ="0.0"> 

<CenterLocation  locationld ="B"  x="50.0"  y="50.0"  /> 
</Parameters> 

</(ttackDispersionParameters> 

<GroupProbe> 

<AttackLocation  locationld ="1"  x="25"  y ="18" 

lat="0. 13025057227020484"  lon=  "0 . 1 80 90357 2597 50 67 "/ > 
<Groups>A  B</Groups> 

</ GroupProbe> 

<TroopSelectionProbe> 

<Groups>A  B</Groups> 

</TroopSelectionProbe> 

<GroundTruth  responsibleGroup= " A "/ > 

<GroundTruthSurpriseProbe  minSurpris e=,,0,r  maxSurprise=  ”6" 
increment^ "1 "/ > 

<TrialResponse  trialTime_ms= "11903 "> 

<ResponseFeedback  responseWellFormed= "true "> 

<GroundTruth  responsibleGroup= "A "/ > 

<ProbabilitiesScore_sl>100 . 0</ProbabilitiesScore_sl> 
<TroopAllocationScore_s2>100 . 0</TroopAllocationScore_s2> 
</ResponseFeedback> 

<GroupResponse  trialPartTime_ms= "3229"> 
<NormativeProbsCumulative> 

50.0  50.0 

</NormativeProbsCumulative> 

<ProbabilitiesScore_sl>100 . 0</ProbabilitiesScore_sl> 
<GroupAttackProbability  groups "A"  probability= "50 " 
time_ms= "89 7 "/> 

<GroupAttackProbability  group ="B"  probability=  "50 ” 
time_ms= "455 "/ > 

</GroupResponse> 

CTroopSelectionResponse  groups "A"  trialPartTime_ms ="2 683"> 
<TroopAllocationScore_s2>100 . 0</TroopAllocationScore_s2> 
<TroopSelectionResponse/> 

<GroundTruthSurpriseResponse  surprise^ ”5 " 
trial Par tTime_ms= "1 982" /> 

</TrialResponse> 

</ProbeTrial> 

</TrialBlock> 

</ExamPhase> 

Figure  18:  Task  1  sample  response 
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4.2  Task  2  Output 


Major  differences:  The  task  2  TrialResponse  element  also  contains  a  GroupCirclesResponse  element, 
which  contains  the  center  locations  and  radii  of  each  group  circle.  Radii  distances  should  be  specified 
in  grid  units  (see  Section  2.8  for  details). 


<!--  Task  2  Response  --> 

<ExamPhase  xsi : type= "ns3 : Task_2_Phase "  examld= "Sample  Exam" 
nam e="M±ss±on  2"  id =”Task2"  startTim e=”2011-07-27T14 : 20 : 45 . 571-04 : 00” 
endTime= "2 011-07 -27T14 : 50: 45. 57 1-04:  00  ”> 

<TrialBlock  xsi : type= ”ns3 : Task_2_TrialBlock ”  trialBlockNum= ”1 ” 
numPresentationTrials= ”1 9 ”> 

<FeatureVectorFile  f eatureVectorUrl_CSV= ”task2_l . csv” 
f eatureVectorUrl_KML= ”task2_l . kml ”/ > 

<ProbeTrial  trialNum =”20”> 

<AttackDispersionParameters> 

<Parameters  baseRate=  ”0 . 25  ”  group="A"  sigma  X=”5.0” 
sigmaY =”5.0"  theta ="0.0"> 

<CenterLocation  x="25.0"  y="50.0"  /> 

</Parameters> 

<Parameters  baseRat e=”0.25"  group ="B"  sigma X=”1.0" 
sigmaY= "1 . 0 "  theta ="0.0"> 

<CenterLocation  x="50.0"  y ="25.0"  /> 

</Parameters> 

<Parameters  baseRat e="0.25"  group="C"  sigma X="9.0" 
sigmaY= "9 . 0 "  theta ="0.0"> 

<CenterLocation  x="75.0"  y ="75.0"  /> 

</Parameters> 

<Parameters  baseRat e="0 . 25"  qroup="D"  sigma X="2.5" 
sigmaY= ”2 . 5 "  theta ="0.0"> 

<CenterLocation  x="10.0"  y ="10.0"  /> 

</Parameters> 

</|ttackDispersionParameters> 

<GroupCirclesProbe> 

<Groups>A  B  C  D</Groups> 

</ GroupCirclesProbe> 

<GroupProbe> 

<AttackLocation  locationld ="1"  x="25"  y ="18" 

±at=” 0 .13025057227020484"  lon= "0 . 1 809035 7259 75067 "/> 
<Groups>A  B  C  D</Groups> 

</ GroupProbe> 

<TroopSelectionProbe> 

<Groups>A  B  C  D</Groups> 

</TroopSelectionProbe> 

<GroundTruth  re sponsible Groups "B " / > 

<GroundTruthSurpriseProbe  minSurpris e="0”  maxSurprise= "6" 
increment^ ”1 "/ > 

<TrialResponse  trialTime_ms= " 2960 6”> 

<ResponseFeedback  responseWellFormed= "true "> 

<GroundTruth  responsibleGroup=  "B "/ > 

<ProbabilitiesScore_sl>100 . 0</ProbabilitiesScore_sl> 
<TroopAllocationScore_s2>100 . 0</TroopAllocationScore_s2> 
</ResponseFeedback> 

<GroupCirclesResponse  trialPartTime_ms= "5698"> 
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<GroupCircle  group =  "A"  radius= "5 . 0 "> 

<CenterLocation  x="3"  y="4"  lat= "0. 028944571 615601076” 
Ion =”0.021 708428711700804 "/> 

</ GroupCircle> 

<GroupCircle  group ="B"  radius ="9.0"> 

<CenterLocation  x=”8"  y="5"  lat =”0.03618071451950134” 
lon="0 . 05788914323120215"/> 

</ GroupCircle> 

<GroupCircle  group =  "C"  radius ="6.0"> 

<CenterLocation  x="8"  y =”12”  lat =”0.08683371484680322" 
lon="0. 05788914323120215" /> 

</ GroupCircle> 

<GroupCircle  group ="D"  radius= "3 . 0 "> 

<CenterLocation  x="6"  y="4”  lat= "0. 028944571 615601076" 
Ion  =”0.04341685742340161  "/> 

</ GroupCircle> 

</ GroupCirclesResponse> 

<GroupResponse  trialPartTime_ms= "15436"> 
<NormativeProbsCumulative> 

25.0  25.0  25.0  25.0 
</NormativeProbsCumulative> 

<ProbabilitiesScore_sl>100 . 0</ProbabiiitiesScore_sl> 
<GroupAttackProbability  groups "A"  probability= ”25 " 
time_ms= ”4533 "/> 

<GroupAttackProbability  group ="B"  probability= ”25 " 
time_ms= "1344 "/ > 

<GroupAttackProbability  group ="C”  probability ="25"/ 
time_ms= "2200 "> 

<GroupAttackProbability  group =”D"  probability= "25 " 
time_ms= ”988 "/ > 

</GroupResponse> 

<TroopSelectionResponse  group ="B"  trialPartTime_ms =”1638"> 

<TroopAllocationScore_s2>100 . 0</TroopAllocationScore_s2> 
<TroopSelectionResponse/> 

<GroundTruthSurpriseResponse  surprise^ ”5 " 
trial Par tTime_ms= ”3229" / > 

</TrialResponse> 

</ProbeTrial> 

</TrialBlock> 

</ExamPhase> 

Figure  19:  Task  2  sample  response 


4.3  Task  3  Output 

Major  differences:  A  task  3  TrialResponse  element  contains  a  GroupCentersResponse  element  instead 
of  a  GroupCirclesResponse  element.  The  GroupCentersResponse  element  contains  the  "center  of 
gravity"  location  for  each  group. 


<!--  Task  3  Response  --> 

<ExamPhase  xsi : type= "ns3 : Task_3_Phase "  examld= "Sample  Exam" 
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name=  "Missi  on  3"  id  ="Task3"  startTime=”2011-07-27T14  ;  50 :  45 . 571-04  :  00" 
endTime= "201 1-0 7-2 7T1 5: 20: 45. 5 71 -04:00 "> 

<RoadsFile  f eatureVectorUrl_CSV= " roads . csv" 
f eatureVectorUrl_KML= "roads . kml "/ > 

<TrialBlock  xsi : type= ”ns3 : Task_3_TrialBlock ”  trialBlockNum= ”1 ” 
numPresentationTrials= "1 9 "> 

<FeatureVectorFile  f eatureVectorUrl_CSV= ”task3_l . csv" 
f eatureVectorUrl_KML= "task3_l . kml "/> 

<ProbeTrial  trialNum ="20 "> 

<|LttackDispersionParameters> 

<Parameters  baseRate= "0 . 25 "  groups "A "> 

<CenterLocation  locationId= "A "  x="10.0"  y ="30.0"  /> 
</Parameters> 

<Parameters  baseRat e="0 . 25"  groups "B"> 

<CenterLocation  locationld ="B"  x="50.0"  y ="25.0"  /> 
</Parameters> 

<Parameters  baseRat e="0 . 25"  group ="C"> 

<CenterLocation  locationld^ "C"  x="75.0"  y ="25.0"  /> 
</Parameters> 

<Parameters  baseRat e=”0 . 25"  group ="D"> 

<CenterLocation  locationld ="D"  x="25.0"  y ="50.0"  /> 
</Parameters> 

</^ttackDispersionParameters> 

<GroupCenters Probe > 

<Groups>A  B  C  D</Groups> 

</ GroupCentersProbe> 

<GroupProbe> 

<AttackLocation  locationld^ "1 "  x="13"  y="37" 
lat="0 .2677372874443099"  lon="0 . 0940 698577507034 9" /> 

<Groups>A  B  C  D</Groups> 

</ GroupProbe> 

<TroopSelectionProbe> 

<Groups>A  B  C  D</Groups> 

</TroopSelectionProbe> 

<GroundTruth  responsibleGroup= "D" / > 

<GroundTruthSurpriseProbe  minSurpris e="0”  maxSurpris e=”6" 
increment^ ”1 "/ > 

<TrialResponse  trialTime_ms= "5443 9"> 

<ResponseFeedback  responseWellFormed= "true "> 

<GroundTruth  responsibleGroup= "D" / > 

<ProbabilitiesScore_sl>100 . 0</ProbabilitiesScore_sl> 
<TroopAllocationScore_s2>100 . 0</TroopAllocationScore_s2> 
</ResponseFeedback> 

<GroupCentersResponse  trialPartTime_ms= "6894 "> 

<GroupCenter  groups "A "> 

<CenterLocation  x= "3 "  y ="4"  lat= "0. 028944571 615601076" 
Ion ="0.021 708428711700804" /> 

</ GroupCenter> 

<GroupCenter  groups  "B"> 

<CenterLocation  x="8"  y ="5"  lat ="0.03618071451950134" 
lon=”0. 057 88914323120215" / > 

</ GroupCenter> 

<GroupCenter  group="C"> 

<CenterLocation  x="8"  y ="12"  lat ="0.08683371484680322" 
Ion ="0 .05788914323120215" /> 

</ GroupCenter> 
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<GroupCenter  group =  "D”> 

<CenterLocation  x="6"  y="4"  lat= "0. 028944571 615601076” 
Ion  ="0.04341685742340161  "/> 

</ GroupCenter> 

</GroupCentersResponse> 

<GroupResponse  trial Par tTime_ms= ”1 712 9 "> 
<NormativeProbsCumulative> 

25.0  25.0  25.0  25.0 
</NormativeProbsCumulative> 

<ProbabilitiesScore_sl>100 . 0</ProbabilitiesScore_sl> 
<GroupAttackProbability  groups "A"  probability= ”25 " 
time_ms= "2344 "/ > 

<GroupAttackProbability  group ="B"  probability=  ”25 " 
time_ms= "2353 "/ > 

<GroupAttackProbability  group ="C"  probability ="25" 
time_ms= ”3836" / > 

<GroupAttackProbability  group ="D"  probability= "25 " 
time_ms= "1 837" / > 

</GroupResponse> 

<TroopSelectionResponse  qroup="D"  trialPartTime_ms ="2418"> 

<TroopAllocationScore_s2>100 . 0</TroopAllocationScore_s2> 
<TroopSelectionResponse/> 

<GroundTruthSurpriseResponse  surprise^ ”5 " 
trial Par tTime_ms= "2106" / > 

</TrialResponse> 

</ProbeTrial> 

</TrialBlock> 

</ExamPhase> 

Figure  20:  Task  3  sample  response 


4.4  Task  4  Output 

Major  differences:  Like  Tasks  1-3,  each  response  in  Task  4  is  contained  within  a  TrialResponse  element. 
An  initial  LocationResponse  element  contains  the  probability  of  attack  at  each  location.  Next,  a 
LocationResponse_afterlNT  element  contains  the  updated  probability  of  attack  at  each  location  after 
being  shown  the  SOCINT  layer.  Finally,  a  TroopAllocationResponse  element  contains  the  participant's 
troop  allocations  across  locations,  and  a  GroundTruthSurpriseResponse  element  contains  the 
participant's  "surprise"  after  being  shown  the  actual  attack  location. 

Normative  Response  Data  and  Scores:  The  initial  LocationResponse  element  contains  a 
NormativeProbsCumulative  element  with  the  the  initial  normative  probabilities  and  a 
ProbabilitiesScore_sl  element  that  contains  the  participant's  SI  score  based  on  the  initial  probability 
assessment.  The  LocationResponse_afterlNT  element  contains  the  updated  normative  probabilities 
and  SI  score.  The  overall  SI  score  for  the  trial  in  the  ResponseFeedback  element  is  the  average  of 
these  two  scores. 


<!--  Task  4  Response  --> 

<ExamPhase  xsiityp e="ns3 : Task_4_Phase"  examld= "Sample  Exam" 
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name: =  "Mis  si  on  4"  id  =”Task4"  startrYlme=" 2011-07 -27T15 ;  20 :  45 . 571-04  :  00" 
endTime= "201 1-0 7-2 7T1 5: 50: 45. 5 71 -04:00  "> 

CTrial  trialNum  ="1"> 

<FeatureVectorFile  f eatureVectorUrl_CSV= " task4_l . csv" 
f eatureVectorUrl_KML= "task4_l . kml "/ > 

<RoadsFile  f eatureVectorUrl_CSV= ” roads . csv" 
f eatureVectorUrl_KML= "roads . kml "/> 

<RegionsFile  f eatureVectorUrl_CSV= "S0CINT_1 . csv" 
f eatureVectorUrl_KML= "S0CINT_1 . kml "/>  ~ 

<LocationProbe> 

<AttackGroup>A</AttackGroup> 

<Locations>l  2  3  4</Locations> 

</LocationProbe> 

<INTLayers> 

<INTLayer> 

<LayerType  xsi : type= ” SoclntLayer" / > 

<LocationProbe> 

<AttackGroup>A</AttackGroup> 

<Locations>l  2  3  4</Locations> 

</LocationProbe> 

<SurpriseProbe  minSurpris e=”0"  maxSurprise= "6"  increment ="1"  /> 
</INTLayer> 

</INTLayers> 

<TroopAllocationProbe> 

<Locations>l  2  3  4</Locations> 

</TroopAllocationProbe> 

<GroundTruth  attackLocationId= "3 "/ > 

<GroundTruthSurpriseProbe  minSurpris e="0"  maxSurprise= "6" 
increment^ "1 " / > 

<TrialResponse  trialTime_ms= "13572 "> 

<ResponseFeedback  responseWellFormed= "true "> 

<GroundTruth  attackLocationId= "3 "/> 

<ProbabilitiesScore_sl>100 . 0</ProbabilitiesScore_sl> 
<TroopAllocationScore_s2>25 . 0</TroopAllocationScore_s2> 
</ResponseFeedback> 

<LocationResponse  Group=rrArr  trialPartTime_ms=  "8938  "> 
<NormativeProbsCumulative> 

25.0  25.0  25.0  25.0 
</NormativeProbsCumulative> 

<ProbabilitiesScore_sl>100 . 0</ProbabiiitiesScore_sl> 
<GroupAttackProbability  locationld ="1"  probability ="25" 
time_ms= "2100" / > 

<GroupAttackProbability  locationId= "2 "  probability ="25" 
time_ms= "1 083 "/ > 

<GroupAttackProbability  locationId= "3 "  probability^ "25 " 
time_ms= "1000" / > 

<GroupAttackProbability  locationld ="4"  probability= "25 " 
time_ms= "2411 "/ > 

</LocationResponse> 

<LocationResponse_af terINT> 

<INTLayerShown  userSelected= "false "> 

<LayerType  xsi : type= "SoclntLayer " /> 

</INTLayerShown> 

<LocationResponse  Groups "A"  trialPartTime_ms= "4353 "> 
<NormativeProbsCumulative> 

25.0  25.0  25.0  25.0 
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</NormativeProbsCumulative> 

<ProbabilitiesScore_sl>100 . 0</ProbabilitiesScore_sl> 
<GroupAttackProbability  locationId= "1 "  probability ="25" 
time_ms= "1 078 "  / > 

<GroupAttackProbability  locationId= "2 "  probability ="25” 
time_ms= "420" /> 

<GroupAttackProbability  locationId= ”3 "  probability= "25 " 
time_ms= "678 "/> 

<GroupAttackProbability  locationId= "4 "  probability^ "25 " 
time_ms= "1 780" / > 

</LocationResponse> 

</LocationResponse_af terINT> 

<TroopAllocationResponse  trial Par tTime_ms= ”5738 ”> 

<TroopAllocationScore_s2>25 . 0</TroopAllocationScore_s2> 
<TroopAllocation  locationld ="1"  allocation ="25" 
time_ms= ”899 "/ > 

<TroopAllocation  locationId= "2 "  allocation=  ”25 " 
time_ms= "400" /> 

CTroopAllocation  locationId= "3 "  allocation^ "25 " 
time_ms= ”678 " / > 

CTroopAllocation  locationld ="4"  allocation =”25" 
time_ms= ”1 780" /> 

</TroopAllocationResponse> 

CGroundTruthSurpriseResponse  surprise^ "3 " 
trial Par tTime_ms= "1840 "/ > 

</TrialResponse> 

</Trial> 

</ExamPhase> 

Figure  21:  Task  4  sample  response 


4.5  Tasks  5-6  Output 

Major  differences:  The  TrialResponse  element  first  contains  an  InitialProbabilities  element,  which 
contains  the  probabilities  that  were  provided  to  the  participant  in  the  InitialHumintReport  element  at 
the  start  of  the  trial.  Next,  a  sequence  of  one  or  more  GroupResponse_afterlNT  elements  contain  the 
updated  probabilities  after  each  INT  layer  is  presented  and  whether  the  INT  layer  was  selected  by  the 
participant  or  not  (only  true  for  Task  6).  Finally,  a  TroopAllocationResponse  element  contains  the 
participant's  troop  allocations  across  groups,  and  a  GroundTruthSurpriseResponse  element  contains 
the  participant's  "surprise"  after  being  shown  the  group  actually  responsible  for  the  attack. 

Normative  Response  Data  and  Scores:  Each  GroupResponse_afterlNT  element  contains  the  normative 
probabilities  based  on  the  INT  layers  that  have  been  observed  and  the  participant's  SI  score  based  on 
their  probability  assessment.  The  overall  SI  score  for  the  trial  in  the  ResponseFeedback  element  is  the 
average  of  these  scores.  For  Task  6,  the  INTLayerExpectedUtilities  element  contains  the  expected 
informatic  utility  of  selecting  each  INT  layer.  The  normative  response  is  to  select  the  layer  with  the 
highest  informatic  utility. 
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<!--  Task  6  Response  --> 

<ExamPhase  xsi  :  type=  ”ns3 :  Task_6_Phase  "  examld  =  "Sample  Exam” 
name= "Mission  6"  id ="Task6" 
startTim e=" 2011-07 -27T16 : 20 : 45 . 571-04 : 00" 
endTime="2 01 1-07-27T1 6: 50: 45. 57 1-04: 00 "> 

<Trial  trialNum= "1 "  numLayersToShow ="3"> 

<FeatureVectorFile  f eatureVectorUrl_CSV= "task6_l . csv" 
f eatureVectorUrl_KML= "task6_l . kml "  /> 

<RoadsFile  f eatureVectorUrl_CSV= "roads . csv" 
f eatureVectorUrl_KML= "roads . kml "  /> 

<RegionsFile  f eatureVectorUrl_CSV= "S0CINT_1 . csv" 
f eatureVectorUrl_KML= "S0CINT_1 . csv"  /> 
<InitialHumintReport> 

<Groups>A  B  C  D</Groups> 

<Probabilities>25  25  25  25</Probabilities> 
</InitialHumintReport> 

<INTLayers> 

<INTLayer> 

<LayerType  xsi : type= "ns3 : ImintLayer"  /> 
<GroupProbe> 

<AttackLocation  locationId= "3 "  /> 
<Groups>A  B  C  D</Groups> 

</ GroupProbe> 

</INTLayer> 

<INTLayer> 

<LayerType  xsi : type= ”ns3 : MovintLayer"  /> 
<GroupProbe> 

<AttackLocation  locationId= ”3 "  /> 
<Groups>A  B  C  D</Groups> 

</ GroupProbe> 

</INTLayer> 

<INTLayer> 

<LayerType  xsi : type= ”ns3 : SigintLayer"> 
<Group>A</ Group> 

</LayerType> 

<GroupProbe> 

<AttackLocation  locationId= "3 "  /> 
<Groups>A  B  C  D</Groups> 

</ GroupProbe> 

</INTLayer> 

<INTLayer> 

<LayerType  xsi : type= "ns3 : SigintLayer"> 
<Group>B</Group> 

</LayerType> 

<GroupProbe> 

<AttackLocation  locationId= "3 "  /> 
<Groups>A  B  C  D</Groups> 

</ GroupProbe> 

</INTLayer> 

<INTLayer> 

<LayerType  xsi : type= ”ns3 : SigintLayer"> 
<Group>C</Group> 

</LayerType> 

<GroupProbe> 

<AttackLocation  locationId= "3 "  /> 
<Groups>A  B  C  D</Groups> 
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</ GroupProbe> 

</INTLayer> 

<INTLayer> 

<LayerType  xsi : type= "ns3 : SigintLayer"> 

<Group>D</ Group> 

</LayerType> 

<GroupProbe> 

<AttackLocation  locationId= "3 "  /> 

<Groups>A  B  C  D</Groups> 

</ GroupProbe> 

</INTLayer> 

<INTLayer> 

<LayerType  xsi : type= "ns3 : SocintLayer"  /> 

<GroupProbe> 

<AttackLocation  locationld =  "3"  /> 

<Groups>A  B  C  D</Groups> 

</ GroupProbe> 

</INTLayer> 

</INTLayers> 

<TroopAllocationProbe> 

<Groups>A  B  C  D</Groups> 

</TroopAllocationProbe> 

<GroundTruth  responsibleGroup ="D”  /> 

<GroundTruthSurpriseProbe  minSurprise= "0 " 
maxSurprise=  "6"  increment  =  "1"  /> 

CTrialResponse  trialTime_ms= " 664 96"> 

<ResponseFeedback  responseWellFormed= "true "> 

<GroundTruth  responsibleGroup^ ”D"  /> 

<ProbabilitiesScore_sl>100 . 0</ProbabilitiesScore_sl> 
<TroopAllocationScore_s2>25 . 0</TroopAllocationScore_s2> 
</ResponseFeedback> 

<InitialProbabilities>25  25  25  25</InitialProbabilities> 
<GroupResponse_af terINT> 

<INTLayerExpectedUtilities> 

<ExpectedUtility 

expectedUtility_cumulative= "0 . 25 "> 
<LayerType  xsirtyp e= ”ns3 : ImintLayer"  /> 
</ExpectedUtility> 

<ExpectedUtility 

expectedUtility_cumulative= "0 . 25 "> 
<LayerType  xsirtyp e= ”ns3 : MovintLayer"  /> 
</ExpectedUtility> 

<ExpectedUtility 

expectedUtility_cumulative= " .25 "> 
<LayerType  xsi : type= "ns3 : SigintLayer"> 
<Group>A</Group> 

</LayerType> 

</ExpectedUtility> 

<ExpectedUtility 

expectedUtility_cumulative= "0 . 25 "> 
<LayerType  xsi : type= ”ns3 : SigintLayer"> 
<Group>B</Group> 

</LayerType> 

</ExpectedUtility> 

<ExpectedUtility 

expectedUtility_cumulative= "0 . 25 "> 
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<LayerType  xsi : type= "ns3 : SigintLayer”> 
<Group>C</Group> 

</LayerType> 

</ExpectedUtility> 

<ExpectedUtility 

expectedUtility_cumulative= ”0 . 25 "> 
<LayerType  xsi : type= ”ns3 : SigintLayer"> 
<Group>D</Group> 

</LayerType> 

</ExpectedUtility> 

<ExpectedUtility 

expectedUtility_cumulative= ”0 . 25 "> 
<LayerType  xsi : type= "ns3 : SocintLayer"  /> 
</ExpectedUtility> 

</INTLayerExpectedUtilities> 

<INTLayerShown  userSelected= "true " 
layer Select ionTime_ms= "23 12 "> 

<LayerType  xsi : type= "ns3 : SigintLayer"> 
<Group>A</ Group> 

</LayerType> 

</INTLayerShown> 

<GroupResponse  locationld =”1" 
trialPartTime_ms= " 6030 "> 
<NormativeProbsCumulative> 

25.0  25.0  25.0  25.0 
</NormativeProbsCumulative> 
<ProbabilitiesScore_sl>100 . 0</ProbabilitiesScore_sl> 
<GroupAttackProbability  groups "A" 

probability^ "25 "  time_ms ="1324"  /> 
<GroupAttackProbability  groups "B " 

probability= "25 "  time_ms= "1 123 "  /> 
<GroupAttackProbability  groups "C" 

probability =”25"  time_ms= "1245 "  /> 
<GroupAttackProbability  groups "D" 

probability^ "25 ”  time_ms =”333"  /> 

</ GroupResponse> 

</ GroupResponse_af terINT> 

<GroupResponse_af terINT> 

<INTLayerExpectedUtilities> 

<ExpectedUtility 

expectedUtility_cumulative= ”0 . 25 "> 
<LayerType  xsi : type= ”ns3 : ImintLayer"  /> 
</ExpectedUtility> 

<ExpectedUtility 

expectedUtility_cumulative= "0 . 25 "> 
<LayerType  xsi : type= "ns3 : MovintLayer"  /> 
</ExpectedUtility> 

<ExpectedUtility 

expectedUtility_cumulative= "0 . 25 "> 
<LayerType  xsiityp e="ns3 : SocintLayer"  /> 
</ExpectedUtility> 

</INTLayerExpectedUtilities> 

<INTLayerShown  userSelected= "true " 
layerSelectionTime_ms= "1 638 "> 

<LayerType  xsi : type= "ns3 : MovintLayer "  /> 
</INTLayerShown> 


42 


<GroupResponse  locationId= "1 " 

trial Par tTime_ms= "12075 "> 
<NormativeProbsCumulative> 

25.0  25.0  25.0  25.0 
</NormativeProbsCumulative> 
<ProbabilitiesScore_sl>100 . 0</ProbabilitiesScore_sl> 
<GroupAttackProbability  groups "A" 

probability ="25"  time_ms= "30 78 "  /> 
<GroupAttackProbability  groups "B " 

probability^ "25 "  time_ms= "3434 "  /> 
<GroupAttackProbability  groups "C” 

probability =”25"  time_ms= "1234 "  /> 
<GroupAttackProbability  groups "D" 

probability^  "25 "  time_ms ="4532"  /> 
</GroupResponse> 

</ GroupResponse_af terINT> 

<GroupResponse_af terINT> 

<INTLayerExpectedUtilities> 

<ExpectedUtility 

expectedUtility_cumulative= "0 . 25 "> 
<LayerType  xsi : type= "ns3 : Imin tLayer "  /> 
</ExpectedUtility> 

<ExpectedUtility 

expectedUtility_cumulative= "0 . 25 "> 
<LayerType  xsiityp e="ns3 : SocintLayer"  /> 
</ExpectedUtility> 

</INTLayerExpectedUtilities> 

<INTLayerShown  userSelected= "true " 
layerSelectionTime_ms= "6084 "> 

<LayerType  xsi : type= "ns3 : Imin tLayer "  /> 
</INTLayerShown> 

<GroupResponse  locationId= "1 " 
trial Par tTime_ms= ”4213"> 

<NormativeProbsCumulative> 

25.0  25.0  25.0  25.0 
</NormativeProbsCumulative> 
<ProbabilitiesScore_sl>100 . 0</ProbabilitiesScore_sl> 
<GroupAttackProbability  groups "A" 

probability^ "25 "  time_ms= "783 "  /> 
<GroupAttackProbability  groups "B " 

probability= "25 "  time_ms= "1323 "  /> 
<GroupAttackProbability  groups "C" 

probability ="25"  time_ms= "34 6"  /> 
<GroupAttackProbability  groups "D" 

probability^  "25 "  time_ms ="467"  /> 

</ GroupResponse> 

</ GroupResponse_af terINT> 

<TroopAllocationResponse  trialPartTime_ms= "7854 "> 

<TroopAllocationScore_s2>25 . 0</TroopAllocationScore_s2> 
<TroopAllocation  groups "A"  allocation ="25" 
time_ms =”345"  /> 

<TroopAllocation  group ="B"  allocation ="25" 
time_ms =”687"  /> 

<TroopAllocation  groups "C"  allocation ="25" 
time_ms ="5677"  /> 

CTroopAllocation  group ="D"  allocation= ”25 " 
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time_ms= "456"  /> 
</TroopAllocationResponse> 

<GroundTruthSurpriseResponse  surprise= ”3 "  /> 
</TrialResponse> 

</Trial> 

Figure  22:  Tasks  5-6  sample  response 


4.6  Task  7  Output 

Major  differences:  Each  TrialResponse  element  contains  a  GroupProbeResponse  element  that  contains 
the  probability  of  attack  by  each  group,  a  LocationProbeResponse  element  that  contains  the 
probability  of  attack  at  each  location,  and  the  TroopAllocationProbeResponse  element  that  contains 
the  troop  allocation  at  each  location.  Note  that  a  GroupProbeResponse  element  should  always  be 
present  even  if  the  participant  does  not  update  group  attack  probabilities.  On  the  first  trial,  the 
GroupProbeResponse  element  should  contain  the  initial  group  attack  probabilities  based  on  the  initial 
responsible  group.  Finally,  a  sequence  of  zero  or  more  INTLayerPurchase  elements  contains  the  INT 
layers  purchased  by  the  participant. 

Normative  Response  Data  and  Scores:  Normative  probabilities  are  not  computed  for  Task  7.  The 
ResponseFeedback  element  contains  the  participant's  S2  score  based  on  their  troop  allocation,  the 
number  of  credits  awarded  based  on  the  S2  score,  the  number  of  credits  used  purchasing  INT  layers, 
and  the  number  of  credits  remaining. 


<!--  Task  7  --> 

<ExamPhase  xs i : type= "ns3 : Task_  7_Phase "  examld= "Sample  Exam" 

nam  e="Mission  7"  id  =  "Task7"r  startTime=  ”201 1  -08-1 1 T12 : 3  7: 51 . 054-04  :  00  ” 
endTime= "201 1 -08-1 1T1 3: 07: 51 . 054-04  :  00"> 

<RoadsFile  f eatureVectorUrl_CSV= "roads . csv” 
f eatureVectorUrl_KML= "roads . kml "/> 

<RegionsFile  f eatureVectorUrl_CSV= "SOCINT_l . csv" 
f eatureVectorUrl_KML= ”SOCINT_l . csv”/> 
<InitialResponsibleGroup>A</InitialResponsibleGroup> 
<InitialCredits>10</InitialCredits> 

<CorrectPredictionCredits>l</ CorrectPredictionCredits> 

CTrial  trialNum= "1 "> 

<FeatureVectorFile  f eatureVectorUrl_CSV= "task7_l . csv" 
f eatureVectorUr 1_KML= "task  7_1 . kml "/ > 

<GroupProbe> 

<Groups>A  B  C  D</Groups> 

</ GroupProbe> 

<LocationProbe> 

<Locations>l  2  3  4</Locations> 

</LocationProbe> 

<TroopAllocationProbe> 

<Locations>l  2  3  4</Locations> 

</TroopAllocationProbe> 

<ResponsibleGroupShown>true</ResponsibleGroupShown> 

<GroundTruth  responsibleGroup= "A ”  attackLocationld ="4"/> 
<INTLayers> 
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CINTLayer  costCredits ="1"> 

<LayerType  xsi : type= "ns3 : ImintLayer" / > 

</INTLayer> 

<INTLayer  costCredits ="1"> 

<LayerType  xsi : type= "ns3 : MovintLayer" / > 

</INTLayer> 

<INTLayer  costCredits= ”2 "> 

<LayerType  xsi : type= "ns3 : SigintLayer"> 

<Group>A</ Group> 

</LayerType> 

</INTLayer> 

<INTLayer  costCredits= ”2 "> 

<LayerType  xsi : type= "ns3 : SigintLayer"> 

<Group>B</ Group> 

</LayerType> 

</INTLayer> 

<INTLayer  costCredits= ”2 "> 

<LayerType  xsi : type= "ns3 : SigintLayer"> 

<Group>C</ Group> 

</LayerType> 

</INTLayer> 

<INTLayer  costCredits= ”2 "> 

<LayerType  xsi : type= "ns3 : SigintLayer"> 

<Group>D</ Group> 

</LayerType> 

</INTLayer> 

<INTLayer  costCredits= ”2 "> 

<LayerType  xsi : type= "ns3 : SocintLayer" / > 

</INTLayer> 

</INTLayers> 

<TrialResponse  trialTime_ms= "451 87 "> 

<ResponseFeedback  responseWellFormed= "true "> 

<GroundTruth  responsibleGroup= "B " 
attackLocationld ="4"  /> 

<NumCreditsAwarded>0 . 25</NumCreditsAwarded> 
<NumCreditsRemaining>7 . 25</NumCreditsRemaining> 
<NumCreditsUsed>3 . 0</NumCreditsUsed> 

<T roopAl 1 o cationS co re_s 2 > 

25.0 

</TroopAllocationScore_s2> 

</ResponseFeedback> 

<GroupResponse  trialPartTime_ms= ”33542 "> 

<GroupAttackProbability  group="A"  probability= ”85 "/> 
<GroupAttackProbability  group ="B"  probability^ "5 "/> 
<GroupAttackProbability  groups "C"  probability^ "5 "/> 
<GroupAttackProbability  group ="D"  probability= ”5 "/> 

</ GroupResponse> 

<LocationResponse  trialPartTime_ms= "6243 "> 

<GroupAttackProbability  locationld ="1"  probability^ "25 " 
time_ms= "456"/ > 

<GroupAttackProbability  locationld ="2"  probability ="25" 
time_ms= ”864 "/ > 

<GroupAttackProbability  locationId= "3 "  probability= ”25 " 
time_ms= "1 783 " / > 

<GroupAttackProbability  locationld^ "4 "  probability^ "25 " 
time  ms ="2456” /> 
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</LocationResponse> 

<TroopAllocationResponse  trialPartTime_ms= ”3433 ”> 

<TroopAllocation  locationld ="1"  allocation= ”25 " 
time_ms= "325 "/ > 

<TroopAllocation  locationId= "2 "  allocation ="25" 
time_ms= "673 "/> 

<TroopAllocation  locationId= "3 "  allocation= ”25 " 
time_ms= ”437 "/> 

<TroopAllocation  locationld ="4"  allocation ="25" 
time_ms= ”855 " / > 

</TroopAllocationResponse> 

<LayerPurchaseTime_ms>57 18</ Layer Pur chaseTime_ms> 
<INTLayerPurchase  costCredits ="1 "> 

<INTLayer  xsi : type= "ns3 : ImintLayer" / > 
</INTLayerPurchase> 

<INTLayerPurchase  costCredits= "2 "> 

<INTLayer  xsi : type= "ns3 : SigintLayer"> 
<Group>C</Group> 

</INTLayer> 

</INTLayerPurchase> 

</TrialResponse> 

</Trial> 

</ExamPhase> 

Figure  23:  Task  7  sample  response 
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5  Appendices 

5.1  Appendix  A:  Example  Exam  File 


<?xml  version ="1.1"  encoding= "UTF-8 "  standalone= "yes "?> 

<ns3 : IcarusEvaluation_CPDl  examTime Stamps "201 1-07-27T12: 31:54. 989-04:00" 
name= "Sample  Exam"  id = "Sample  Exam"  xmlns : ns2  = " IcarusCPD_Base " 
xmlns : ns3= ” IcarusCPD_l " 

xmlns : xsi= "http : //www. w3 . org/2001/XMLSchema-instance "> 

<!--  Grid  size,  scale,  and  location  information  --> 

<GridSize  gridWidth= "1 00 "  gridHeight= "1 00 "  milesPerGridUnit= "0 . 2 " 
bottomLef tLat= "0 .  0 "  bottomLef tLon= "0 . 0 "  /> 

<!--  Task  1  --> 

<ExamPhase  xsiityp e="ns3 : Task_l_Phase"  examld =" Sample  Exam" 
name= "Mission  1"  id  =  ”Taskl"> 

<TrialBlock  xsi : type= "ns3 : Task_l_TrialBlock "  trialBlockNum=  "1 " 
numPresentationTrials= "9 "> 

<FeatureVectorFile  f eatureVectorUrl_CSV= "taskl_l . csv" 
f eatureVectorUrl_KML= ”taskl_l .  kml "  /> 

<ProbeTrial  trialNum =”10"> 

<GroupProbe> 

<AttackLocation  locationld ="1"  x="25"  y ="18" 
lat="0 .13025057227020484" 

Ion ="0.1809035725975067"  /> 

<Groups>A  B</Groups> 

</ GroupProbe> 

<TroopSelectionProbe> 

<Groups>A  B</Groups> 

</TroopSelectionProbe> 

<GroundTruth  responsibleGroup= "A "  /> 
<GroundTruthSurpriseProbe  minSurprise= " 0 " 
maxSurpris e="6"  increment ="1"  /> 
</ProbeTrial> 

</TrialBlock> 

</ExamPhase> 

<!--  Task  2  --> 

<ExamPhase  xsiityp e="ns3 : Task_2_Phase"  examld =" Sample  Exam" 
nam e= "Mission  2"  id= ”Task2"> 

CTrialBlock  xsi : type= "ns3 : Task_2_TrialBlock "  trialBlockNum= "1 " 
numPresentationTrials= "1 9 "> 

<FeatureVectorFile  f eatureVectorUrl_CSV= ”task2_l . csv" 
f eatureVectorUrl_KML= "task2_l . kml "  /> 

<ProbeTrial  trialNum ="20"> 

<GroupCircles Probe > 

<Groups>A  B  C  D</Groups> 

</ GroupCirclesProbe> 

<GroupProbe> 

<AttackLocation  locationld ="1"  x="25"  y ="18" 
lat="0 .13025057227020484" 

Ion ="0.1809035725975067"  /> 

<Groups>A  B  C  D</Groups> 

</ GroupProbe> 
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<|TroopSelectionProbe> 

<Groups>A  B  C  D</Groups> 
</TroopSelectionProbe> 

<GroundTruth  responsibleGroup =  "B"  /> 
<GroundTruthSurpriseProbe  minSurprise= " 0  " 
maxSurprise= " 6"  increment ="1"  /> 
</ProbeTrial> 

</TrialBlock> 

</ExamPhase> 

< ! —  Task  3  — > 

<ExamPhase  xsi : type= "ns3 : Task_3_Phase "  examld= "Sample  Exam" 
nam e= "Mission  3"  id  =  "Task3/'> 

<RoadsFile  f eatureVectorUrl_CSV= "roads . csv" 

f eatureVectorUrl_KML= "roads . kml "  /> 

CTrialBlock  xsi : type= "ns3 : Task_3_TrialBlock "  trialBlockNum= "1 " 
numPresentationTrials= "1 9 "> 

<FeatureVectorFile  f eatureVectorUrl_CSV= "task3_l . csv" 
f eatureVectorUrl_KML= "task3_l . kml "  /> 

<ProbeTrial  trialNum ="20 "> 

<GroupCentersProbe> 

<Groups>A  B  C  D</Groups> 

</ GroupCentersProbe> 

<GroupProbe> 

<AttackLocation  locationld ="1"  x="13"  y ="37" 
lat="0 .2677372874443099" 

Ion ="0.09406985775070349"  /> 

<Groups>A  B  C  D</Groups> 

</ GroupProbe> 

<|TroopSelectionProbe> 

<Groups>A  B  C  D</Groups> 
</TroopSelectionProbe> 

<GroundTruth  responsibleGroup =”D"  /> 
<GroundTruthSurpriseProbe  minSurprise= "0 " 
maxSurpris e="6"  increment ="1"  /> 
</ProbeTrial> 

</TrialBlock> 

</ExamPhase> 

<!--  Task  4  --> 

<ExamPhase  xsi : type= "ns3 : Task_4_Phase "  examld = "Sample  Exam" 
nam e= "Mission  4"  id =”Task4"> 

<Trial  trialNum  =”1”> 

<FeatureVectorFile  f eatureVectorUrl_CSV= "task4_l . csv" 
f eatureVectorUrl_KML= "task4_l . kml "  /> 

<RoadsFile  f eatureVectorUrl_CSV= "roads . csv" 
f eatureVectorUrl_KML= "roads . kml "  /> 

<RegionsFile  f eatureVectorUrl_CSV= " S0CINT_1 . csv" 
f eatureVectorUrl_KML=  "S0CINT_1 .  kml  ff7> 
<LocationProbe> 

<AttackGroup>A</AttackGroup> 

<Locations>l  2  3  4</Locations> 

</LocationProbe> 

<INTLayers> 

<INTLayer> 

<LayerType  xsi : type= ”ns3 : SocintLayer"  /> 
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<LocationProbe> 

<AttackGroup>A</AttackGroup> 
<Locations>l  2  3  4</Locations> 
</LocationProbe> 

</INTLayer> 

</INTLayers> 

<TroopAllocationProbe> 

<Locations>l  2  3  4</Locations> 
</TroopAllocationProbe> 

<GroundTruth  attackLocationId=  "3 "  /> 
<GroundTruthSurpriseProbe  minSurprise= ”0  " 
maxSurprisew^"  increment=rrl /> 

</Trial> 

</ExamPhase> 

<!--  Task  5  --> 

<ExamPhase  xsi : type= ”ns3 : Task_5_Phase "  examld = "Sample  Exam" 
nam e= "Mission  5"  id= ”Task5"> 

<Trial  trialNum  ="1"> 

<FeatureVectorFile  f eatureVectorUrl_CSV= "task5_l . csv" 
f eatureVectorUrl_KML= ”task5_l . kml "  /> 

<RoadsFile  f eatureVectorUrl_CSV= "roads . csv” 
f eatureVectorUrl_KML= "roads . kml "  /> 

<RegionsFile  f eatureVectorUrl_CSV= "S0CINT_1 . csv” 
f eatureVectorUrl_KML= ”S0CINT_1 . kml "/> 
<InitialHumintReport> 

<Groups>A  B  C  D</Groups> 

<Probabilities>25  25  25  25</Probabilities> 
</InitialHumintReport> 

<INTLayers> 

<INTLayer> 

<LayerType  xsi : type= "ns3 : Imln tLayer "  /> 
<GroupProbe> 

<AttackLocation  locationld ="1"  /> 
<Groups>A  B  C  D</Groups> 

</ GroupProbe> 

</INTLayer> 

<INTLayer> 

<LayerType  xsi : type= "ns3 :Movln tLayer "  /> 
<GroupProbe> 

<AttackLocation  locationId= "1 "  /> 
<Groups>A  B  C  D</Groups> 

</ GroupProbe> 

</INTLayer> 

<INTLayer> 

<LayerType  xsi : type= ”ns3 : SiglntLayer"> 
<Group>B</Group> 

</LayerType> 

<GroupProbe> 

<AttackLocation  locationld ="1"  /> 
<Groups>A  B  C  D</Groups> 

</ GroupProbe> 

</INTLayer> 

<INTLayer> 

<LayerType  xsi : type= "ns3 : Socln tLayer "  /> 
<GroupProbe> 
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<AttackLocation  locationld ="1"  /> 
<Groups>A  B  C  D</Groups> 

</ GroupProbe> 

</INTLayer> 

</INTLayers> 

<TroopAllocationProbe> 

<Groups>A  B  C  D</Groups> 

</TroopAllocationProbe> 

<GroundTruth  responsibleGroup= "C"  /> 
<GroundTruthSurpriseProbe  minSurprise= "0 " 
maxSurprise=,,6’"  increment=rrl /> 

</Trial> 

</ExamPhase> 

<!--  Task  6  --> 

<ExamPhase  xs i : type= ”ns3 : Task_6_Phase ”  examl d= "Sample  Exam" 
name= "Mis si  on  6"  id =  "Task6"> 

<Trial  trialNum ="1"  numLayersToShow ="3”> 

<FeatureVectorFile  f eatureVectorUrl_CSV= "task6_l . csv" 
f eatureVectorUrl_KML= "task6_l . kml "  /> 

<RoadsFile  f eatureVectorUrl_CSV= "roads . csv" 
f eatureVectorUrl_KML= "roads . kml "  /> 

<RegionsFile  f eatureVectorUrl_CSV= " S0CINT_1 . csv" 
f eatureVectorUrl_KML= "S0CINT_1 . kml "/> 
<InitialHumintReport> 

<Groups>A  B  C  D</Groups> 

<Probabilities>25  25  25  25</Probabilities> 
</InitialHumintReport> 

<INTLayers> 

<INTLayer> 

<LayerType  xsirtyp e= ”ns3 : ImlntLayer"  /> 
<GroupProbe> 

<AttackLocation  locationId= "3 "  /> 
<Groups>A  B  C  D</Groups> 

</ GroupProbe> 

</INTLayer> 

<INTLayer> 

<LayerType  xsirtyp e= ”ns3 : MovlntLayer"  /> 
<GroupProbe> 

<AttackLocation  locationId= "3 "  /> 
<Groups>A  B  C  D</Groups> 

</ GroupProbe> 

</INTLayer> 

<INTLayer> 

<LayerType  xsi : type= "ns3 : SigintLayer"> 
<Group>A</Group> 

</LayerType> 

<GroupProbe> 

<AttackLocation  locationId= "3 "  /> 
<Groups>A  B  C  D</Groups> 

</ GroupProbe> 

</INTLayer> 

<INTLayer> 

<LayerType  xsi : type= "ns3 : SigintLayer"> 
<Group>B</ Group> 

</LayerType> 
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<GroupProbe> 

<AttackLocation  locationId= "3 "  /> 
<Groups>A  B  C  D</Groups> 

</ GroupProbe> 

</INTLayer> 

<INTLayer> 

<LayerType  xsi : type= "ns3 : SigintLayer"> 
<Group>C</Group> 

</LayerType> 

<GroupProbe> 

<AttackLocation  locationId= "3 "  /> 
<Groups>A  B  C  D</Groups> 

</ GroupProbe> 

</INTLayer> 

<INTLayer> 

<LayerType  xsi : type= "ns3 : SigintLayer"> 
<Group>D</Group> 

</LayerType> 

<GroupProbe> 

<AttackLocation  locationId= "3 "  /> 
<Groups>A  B  C  D</Groups> 

</ GroupProbe> 

<SurpriseProbe  minSurpris e="0”  maxSurpris e="6" 
increment  ="l,r  /> 

</INTLayer> 

<INTLayer> 

<LayerType  xsi : type= "ns3 : SocintLayer"  /> 
<GroupProbe> 

<AttackLocation  locationId= "3 "  /> 
<Groups>A  B  C  D</Groups> 

</ GroupProbe> 

</INTLayer> 

</INTLayers> 

<TroopAllocationProbe> 

<Groups>A  B  C  D</Groups> 

</TroopAllocationProbe> 

<GroundTruth  responsibleGroup =”D"  /> 
<GroundTruthSurpriseProbe  minSurprise= ”0 ” 
maxSurprise=  "6,r  increment  ="1"  /> 

</Trial> 

</ExamPhase> 

<!--  Task  7  --> 

<ExamPhase  xsiityp e="ns3 : Task_7_Phase"  examld =" Sample  Exam" 
name= "Mission  7"  id= "Task 7"> 

<RoadsFile  f eatureVectorUrl_CSV= "roads . csv" 
f eatureVectorUrl_KML= "roads . kml "/ > 

<RegionsFile  f eatureVectorUrl_CSV= "S0CINT_1 . csv" 
f eatureVectorUrl_KML= "S0CINT_1 . csv"/>~ 
<InitialResponsibleGroup>A</InitialResponsibleGroup> 
<InitialCredits>10</InitialCredits> 

<CorrectPredictionCredits>l</ CorrectPredictionCredits> 

<Trial  trialNum  ="1”> 

<FeatureVectorFile  f eatureVectorUrl_CSV= " task! _1 . csv" 
f eatureVectorUr 1_KML= "task  7_1 . kml "/ > 

<GroupProbe> 
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<Groups>A  B  C  D</Groups> 

</ GroupProbe> 

<LocationProbe> 

<Locations>l  2  3  4</Locations> 

</LocationProbe> 

<TroopAllocationProbe> 

<Locations>l  2  3  4</Locations> 

</TroopAllocationProbe> 

<ResponsibleGroupShown>true</ResponsibleGroupShown> 
<GroundTruth  responsibleGroup= "A "  attackLocationId= "4 "/> 
<INTLayers> 

<INTLayer  costCredits =  ”1"> 

<LayerType  xsi : type= "ns3 : ImintLayer" / > 

</INTLayer> 

<INTLayer  costCredits ="1"> 

<LayerType  xsi : type= "ns3 :  MovintLayer"  / > 
</INTLayer> 

<INTLayer  costCredits= "2 "> 

<LayerType  xsi : type= "ns3 : SigintLayer"> 

<Group>A</ Group> 

</LayerType> 

</INTLayer> 

<INTLayer  costCredits= "2 "> 

<LayerType  xsi : type= "ns3 : SigintLayer"> 

<Group>B</ Group> 

</LayerType> 

</INTLayer> 

<INTLayer  costCredits= "2 "> 

<LayerType  xsi : type= "ns3 : SigintLayer"> 

<Group>C</ Group> 

</LayerType> 

</INTLayer> 

<INTLayer  costCredits= "2 ”> 

<LayerType  xsi : type= "ns3 : SigintLayer"> 

<Group>D</ Group> 

</LayerType> 

</INTLayer> 

<INTLayer  costCredits= ”2 "> 

<LayerType  xsi : type= "ns3 : SocintLayer" / > 
</INTLayer> 

</INTLayers> 

</Trial> 

<Trial  trialNum= "2 "> 

<FeatureVec tor File  f eatureVectorUrl_CSV= ” task 1_2 . csv" 
f eatureVectorUr 1_KML= "task  7_2 . kml "  /> 

<GroupProbe> 

<Groups>A  B  C  D</Groups> 

</ GroupProbe> 

<LocationProbe> 

<Locations>l  2  3  4</Locations> 

</LocationProbe> 

<TroopAllocationProbe> 

<Locations>l  2  3  4</Locations> 
</TroopAllocationProbe> 

<ResponsibleGroupShown>true</ResponsibleGroupShown> 
<GroundTruth  responsibleGroup= "A "  attackLocation!d= "3 "  /> 
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</Trial> 

</ExamPhase> 

</ns3 : IcarusEvaluation  CPD1> 
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5.2  Appendix  B:  Example  Output 


<?xml  version =”1.0"  encoding ="UTF-8"  standalone^ "yes "?> 

<ns3 : IcarusEvaluation_CPDl  xmlns : ns2  = " IcarusCPD_Base " 
xmlns : ns3= ” IcarusCPD_l " 

xsi : schemaLocation= ” IcarusCPD_Base  IcarusBaseSchema . xsd  IcarusCPD_l 
IcarusCPD_l_Schema . xsd" 

xmlns : xsi= "http : //www. w3 . org/2001/XMLSchema-instance " 
examTimeStamp= ”201 1-07-27T12: 50: 45.571-04:00" 
name= "Sample  Exam"  id =" Sample  Exam" 
startTim e=" 2011-07 -27T13 : 50 : 45 . 571-04 : 00" 
endTim e=" 2011-07 -27T16 : 50 : 45 . 571-04 : 00"> 

<!--  ICArUS  System  information  --> 

<ResponseGenerator> 

<HumanSub j  ect>f alse</HumanSub j  ect> 

<Respons eGene ra tor Id>Model2< /Response Genera tor Id> 

<SiteId>Teaml</ Siteld> 

</ResponseGenerator> 

<!--  Grid  size,  scale,  and  location  information  --> 

<GridSize  gridWidth= "1 00 "  gridHeight= "1 00 "  milesPerGridUnit= "0 . 2 " 
bottomLef tLat= "0 . 0 "  bottomLef tLon ="0.0"  /> 

<!--  Task  1  Response  --> 

<ExamPhase  xsi : type=  ”ns3 : Task_l_Phase "  examld = "Sample  Exam" 
name= "Mission  1"  id ="Taskl" 
startTim e=" 2011-07 -27T13 : 50 : 45 . 571-04 : 00" 
endTime= "201 1-07 -27T1 4:20:45. 571-04:00 ”> 

<TrialBlock  xsi : type= "ns3 : Task_l_TrialBlock "  trialBlockNum= "1 " 
numPresentationTrials= ”9 "> 

<FeatureVectorFile  f eatureVectorUrl_CSV= "taskl_l . csv" 
f eatureVectorUrl_KML= "taskl_l . kml "  /> 

<ProbeTrial  trialNum =”10"> 

<AttackDispersionParameters> 

<Parameters  baseRate= "0 . 5 "  groups "A"  sigmaX= "5 . 0 ” 
sigmaY ="5.0"  theta ="0.0"> 

<CenterLocation  locationId= "A "  x="25.0" 
y="75.0"  /> 

</Parameters> 

<Parameters  baseRate= "0 . 5 "  group ="B"  siqmaX="2.5" 
siqmaY="2 . 5"  theta ="0.0"> 

<CenterLocation  locationld ="B"  x="50.0" 
y= "50 . 0 "  /> 

</Parameters> 

</AttackDispersionParameters> 

<GroupProbe> 

<AttackLocation  locationld ="1"  x="25”  y ="18" 
lat="0 . 13025057227020484" 

Ion ="0.1809035725975067"  /> 

<Groups>A  B</Groups> 

</ GroupProbe> 

<TroopSelectionProbe> 

<Groups>A  B</Groups> 

</TroopSelectionProbe> 
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<GroundTruth  responsibleGroup= "A "  /> 
<GroundTruthSurpriseProbe  minSurprise= "  0 " 
maxSurprise= " 6"  increment ="1"  /> 

CTrialResponse  trialTime_ms= "1 1903 "> 

<ResponseFeedback  responseWellFormed= "true "> 
<GroundTruth  responsibleGroup= "A "  /> 
<ProbabilitiesScore_sl>100 . 0</ProbabilitiesScore_sl> 
<TroopAllocationScore_s2>100 . 0</TroopAllocationScore_s2> 
< /Response Feedbacks 

<GroupResponse  trialPartTime_ms= "3229"> 
<NormativeProbsCumulative>50 . 0 

50 . 0</NormativeProbsCumulative> 
<ProbabilitiesScore_sl>100 . 0 
</ProbabilitiesScore_sl> 
<GroupAttackProbability  groups "A" 

probability =”50"  time_ms= "89 7 "  /> 
<GroupAttackProbability  groups "B " 

probability= ”50 "  time_ms= ”455 "  /> 
</GroupResponse> 

<TroopSelectionResponse  groups "A" 
trialPartTime_ms= "2683 "> 

<T roopAl 1 o cationS co re_s 2 > 

100 . 0</TroopAllocationScore_s2> 
</TroopSelectionResponse> 

<GroundTruthSurpriseResponse  surpris e="5 " 
trialPartTime_ms= "1 982 "  /> 

</TrialResponse> 

</ProbeTrial> 

</TrialBlock> 

</ExamPhase> 

<!--  Task  2  Response  --> 

<ExamPhase  xsi : type= "ns3 : Task_2_Phase "  examld= "Sample  Exam" 
nam e=”M±ssion  2"  id =”Task2" 
startTim e=" 2011-07 -27T14 : 20 : 45 . 571-04  :  00" 
endTim e=" 2011-07 -27T14 : 50 : 45 . 571-04 : 00"> 

CTrialBlock  xsi : type= "ns3 : Task_2_TrialBlock "  trialBlockNum= "1 " 
numPresentationTrials= "1 9 "> 

<FeatureVectorFile  f eatureVectorUrl_CSV= "task2_l . csv" 
f eatureVectorUrl_KML= "task2_l . kml "  /> 

<ProbeTrial  trialNum=  "20 "> 

<AttackDispersionParameters> 

<Parameters  baseRat e="0.25"  group="A"  sigmaX= "5 . 0 " 
sigmaY ="5.0"  theta ="0.0"> 

<CenterLocation  x="25.0"  y ="50.0"  /> 
</Parameters> 

<Parameters  baseRat e="0.25”  group ="B”  sigma X="1.0" 
sigmaY ="1.0"  theta ="0.0”> 

<CenterLocation  x="50 . 0"  y="25.0"  /> 
</Parameters> 

<Parameters  baseRat e="0.25”  group="C"  sigmaX= "9 . 0 " 
sigmaY ="9.0"  theta =”0.0"> 

<CenterLocation  x=”75.0"  y ="75.0”  /> 
</Parameters> 

<Parameters  baseRat e="0 . 25"  qroup="D"  sigmaX= "2 . 5 " 
sigmaY= "2 . 5 "  theta =”0.0"> 
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<CenterLocation  x="10.0"  y ="10.0"  /> 
</Parameters> 

</AttackDispersionParameters> 

<GroupCirclesProbe> 

<Groups>A  B  C  D</Groups> 

</ GroupCirclesProbe> 

<GroupProbe> 

<AttackLocation  locationld =  "1"  x=”25"  y ="18" 
lat=" 0.1302505722702 04 84" 

Ion =”0.1809035725975067"  /> 

<Groups>A  B  C  D</Groups> 

</ GroupProbe> 

<TroopSelectionProbe> 

<Groups>A  B  C  D</Groups> 

</TroopSelectionProbe> 

<GroundTruth  responsibleGroup  =r,B"  /> 
<GroundTruthSurpriseProbe  minSurprise= "0 " 
maxSurprise= "6"  increment ="1"  /> 

<TrialResponse  trialTime_ms= "2 960 6"> 

<ResponseFeedback  responseWellFormed= "true "> 
<GroundTruth  responsibleGroup^ "B "  /> 
<ProbabilitiesScore_sl> 

100 . 0</ProbabiiitiesScore_sl> 

<T roopAl 1 o cationS co re_s 2 > 

100 . 0</TroopAllocationScore_s2> 
</ResponseFeedback> 

<GroupCirclesResponse  trial Par tTime_ms= "5698 "> 
<GroupCircle  group=rrArf  radius  ="5.0" 
time_ms= "1255 "> 

<CenterLocation  x="3"  y ="4" 

lat="0 . 028944571615601076" 

Ion ="0.021708428711700804"  /> 
</ GroupCircle> 

<GroupCircle  groups "B"  radius ="9.0" 
time_ms= "985 "> 

<CenterLocation  x="8"  y =”5" 

lat=" 0 . 03618071451950134" 

Ion ="0.05788914323120215"  /> 
</ GroupCircle> 

<GroupCircle  group ="C"  radius ="6.0" 
time_ms= ”1356"> 

<CenterLocation  x="8"  y ="12" 

lat="0 .08683371484680322" 

Ion ="0.05788914323120215"  /> 
</ GroupCircle> 

<GroupCircle  group ="D"  radius ="3.0" 
time_ms= "2201 "> 

<CenterLocation  x="6"  y ="4" 

lat="0. 028944571 615601076" 

Ion ="0.04341685742340161"  /> 
</ GroupCircle> 

</ GroupCirclesResponse> 

<GroupResponse  trialPartTime_ms= ”15436"> 
<NormativeProbsCumulative> 

25.0  25.0  25.0  25.0 
</NormativeProbsCumulative> 
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<ProbabilitiesScore_sl> 

100 . 0</ProbabilitiesScore_sl> 
<GroupAttackProbability  groups " A” 

probability ="25"  time_ms= ”4533 ”  /> 
<GroupAttackProbability  groups "B ” 

probability =”25”  time_ms= ”1344 ”  /> 
<GroupAttackProbability  groups ”C” 

probability =”25”  time_ms =”2200”  /> 
<GroupAttackProbability  groups  "D,r 

probability^  "25  "  time_ms  ="988,r  /> 
</ GroupResponse> 

<TroopSelectionResponse  groups "B " 
trialPartTime_ms= "1 638"> 

<T roopAl 1 o cationS co re_s 2 > 

100 . 0</TroopAllocationScore_s2> 
</TroopSelectionResponse> 

<GroundTruthSurpriseResponse  surprise= "5 " 
trialPartTime_ms= "3229 "  /> 
</TrialResponse> 

</ProbeTrial> 

</TrialBlock> 

</ExamPhase> 

<!--  Task  3  Response  --> 

<ExamPhase  xsiityp e="ns3 :  Task_3_Phase"  examl d= "Sample  Exam" 
name= "Mission  3"  id =”Task3" 
startTim e="2011-07-27T14 : 50 : 45 . 571-04  :  00" 
endTim e="2011-07-27T15 : 20 : 45 . 571-04 : 00"> 

<RoadsFile  f eatureVectorUrl_CSV= "roads . csv" 
f eatureVectorUrl_KML= "roads . kml "  /> 

<TrialBlock  xsi : type= "ns3 : Task_3_TrialBlock "  trialBlockNum=  "1 " 
numPresentationTrials= "1 9 "> 

<FeatureVectorFile  f eatureVectorUrl_CSV= "task3_l . csv" 
f eatureVectorUrl_KML= "task3_l . kml "  /> 

<ProbeTrial  trialNum ="20 "> 

<AttackDispersionParameters> 

<Parameters  baseRat e="0.25"  group=rrArr> 

<CenterLocation  locationId= "A "  x="10.0" 
y ="30.0"  /> 

</Parameters> 

<Parameters  baseRat e="0.25"  group =”B"> 

<CenterLocation  locationld ="B"  x="50.0" 
y= ”25 . 0 "  /> 

</Parameters> 

<Parameters  baseRat e="0 . 25"  groups "C"> 

<CenterLocation  locationId= "C"  x="75.0" 
y=  ”25 . 0 "  /> 

</Parameters> 

<Parameters  baseRat e="0 . 25"  qroup="D"> 

<CenterLocation  locationld ="D"  x="25.0" 
y-  "50 . 0 "  /> 

</Parameters> 

</AttackDispersionParameters> 

<GroupCentersProbe> 

<Groups>A  B  C  D</Groups> 

</ GroupCentersProbe> 
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<GroupProbe> 

<AttackLocation  locationld ="1"  x=”13"  y="37 " 
lat="0 .2677372874443099" 

Ion ="0.09406985775070349"  /> 

<Groups>A  B  C  D</Groups> 

</ GroupProbe> 

<TroopSelectionProbe> 

<Groups>A  B  C  D</Groups> 

</TroopSelectionProbe> 

<GroundTruth  responsibleGroup ="D"  /> 
<GroundTruthSurpriseProbe  minSurprise= " 0 " 
maxSurprise= "6"  increment ="1”  /> 

<TrialResponse  trialTime_ms= "5443 9"> 

<ResponseFeedback  responseWellFormed= "true "> 
<GroundTruth  responsibleGroup^ "D"  /> 
<ProbabilitiesScore_sl> 

100 . 0</ProbabilitiesScore_sl> 

<T roopAl 1 o cationS co re_s 2 > 

100 . 0</TroopAllocationScore_s2> 
</ResponseFeedback> 

<GroupCentersResponse  trialPartTime_ms= "6894 "> 
<GroupCenter  groups "A"  time_ms= "98 7 "> 
<CenterLocation  x="3"  y ="4" 

lat="0 .028944571615601076" 

Ion ="0.021708428711700804"  /> 
</ GroupCenter> 

<GroupCenter  group ="B”  time_ms ="2256"> 
<CenterLocation  x="8”  y =”5" 

lat=" 0 .03618071451950134" 

Ion ="0.05788914323120215"  /> 
</ GroupCenter> 

<GroupCenter  groups "C"  time_ms= "59 7 ”> 
<CenterLocation  x="8"  y ="12" 

lat=" 0 .08683371484680322" 

Ion ="0.05788914323120215"  /> 
</ GroupCenter> 

<GroupCenter  group ="D"  time_ms= "1 493 "> 
<CenterLocation  x="6”  y ="4" 

lat=" 0 .028944571615601076" 

Ion ="0.04341685742340161"  /> 
</ GroupCenter> 

</ GroupCentersResponse> 

<GroupResponse  trial Par tTime_ms= ”1 712 9 "> 
<NormativeProbsCumulative> 

25.0  25.0  25.0  25.0 
</NormativeProbsCumulative> 
<ProbabilitiesScore_sl> 

100 . 0</ProbabilitiesScore_sl> 
<GroupAttackProbability  groups "A" 

probability^ "25 "  time_ms ="2344"  /> 
<GroupAttackProbability  groups "B " 

probability ="25"  time_ms= "2353 "  /> 
<GroupAttackProbability  groups "C” 

probability^ "25 "  time_ms= "3836"  /> 
<GroupAttackProbability  groups "D" 

probability =”25"  time_ms= "1 83 7"  /> 
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</GroupResponse> 

<TroopSelectionResponse  groups ”D" 
trialPartTime_ms= "2418 "> 

<T roopAl 1 o cationS co re_s 2 > 

100 . 0</TroopAllocationScore_s2> 
</TroopSelectionResponse> 

<GroundTruthSurpriseResponse  surprise=  "5 " 
trialPartTime_ms= "21 06"  /> 

</TrialResponse> 

</ProbeTrial> 

</TrialBlock> 

</ExamPhase> 

<!--  Task  4  Response  --> 

<ExamPhase  xsiityp e="ns3 : Task_4_Phase"  examl d= "Sample  Exam" 
nam  e= "Mission  4"  id  =''  Task4" 
startTime= ”2011-07-27T15:20: 45.571-04:00" 
endTime= "201 1-0 7-2 7T1 5: 50: 45. 5 71 -04:00 "> 

CTrial  trialNum  ="1"> 

<FeatureVectorFile  f eatureVectorUrl_CSV= "task4_l . csv" 
f eatureVectorUrl_KML= ”task4_l . kml "  /> 

<RoadsFile  f eatureVectorUrl_CSV= "roads . csv" 
f eatureVectorUrl_KML= "roads . kml "  /> 

<RegionsFile  f eatureVectorUrl_CSV= "SOCINT_l . csv” 
f eatureVectorUrl_KML= "SOCINT_l . kml "  /> 

<LocationProbe> 

<AttackGroup>A</AttackGroup> 

<Locations>l  2  3  4</Locations> 

</LocationProbe> 

<INTLayers> 

<INTLayer> 

<LayerType  xsiityp e= ”ns3 : SocintLayer”  /> 
<LocationProbe> 

<AttackGroup>A</AttackGroup> 

<Locations>l  2  3  4</Locations> 

</LocationProbe> 

</INTLayer> 

</INTLayers> 

<TroopAllocationProbe> 

<Locations>l  2  3  4</Locations> 

</TroopAllocationProbe> 

<GroundTruth  attackLocationId= "3 "  /> 

<GroundTruthSurpriseProbe  minSurprise= ”0  " 
maxSurprise= " 6"  increment ="1"  /> 

<TrialResponse  trialTime_ms= "13572 "> 

<ResponseFeedback  responseWellFormed= "true "> 

<GroundTruth  attackLocationId= ”3 "  /> 

<ProbabilitiesScore_sl>100 . 0</ProbabiiitiesScore_sl> 
<T roopAl 1 o cationS co re_s 2 > 

25 . 0</TroopAllocationScore_s2> 

</ResponseFeedback> 

<LocationResponse  Groups "A"  trialPartTime_ms= "8938 ”> 
<NormativeProbsCumulative> 

25.0  25.0  25.0  25.0 
</NormativeProbsCumulative> 

<ProbabilitiesScore  sl>100 . 0</ProbabiiitiesScore  sl> 
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<GroupAttackProbability  locationId= "1  " 

probability=  "25 "  time_ms ="2100"  /> 
<GroupAttackProbability  locationId= "2 " 

probability^ "25 "  time_ms= "1 083 "  /> 
<GroupAttackProbability  locationId= " 3 ” 

probability ="25"  time_ms ="1000"  /> 
<GroupAttackProbability  locationId= "4 " 

probability= "25 "  t ime_ms= "241 1 ”  /> 
</LocationResponse> 

<LocationResponse_af terINT> 

<INTLayerShown  userSelected= "false "> 

<LayerType  xsirtyp e= ”ns3 : SocintLayer"  /> 
</INTLayerShown> 

<LocationResponse  Groups "A"  trialPartTime_ms ="4353"> 
<NormativeProbsCumulative> 

25.0  25.0  25.0  25.0 
</NormativeProbsCumulative> 
<ProbabilitiesScore_sl> 

100 . 0</ProbabiiitiesScore_sl> 
<GroupAttackProbability  locationld ="1 " 

probability= ”25 "  time_ms= "1 0 78 ”  /> 
<GroupAttackProbability  locationId= "2 " 
probability=  "25 "  time_ms= " 420 ”  /> 
<GroupAttackProbability  locationId= "3 " 
probability^ ”25 "  time_ms ="334"  /> 
<GroupAttackProbability  locationld^ "4 " 
probability=  "25 ”  time_ms= ”678 ”  /> 
</LocationResponse> 

</LocationResponse_af terINT> 

<TroopAllocationResponse  trial Par tTime_ms= "5 738 ”> 

<T roopAl 1 o cationS co re_s 2 > 

25 . 0</TroopAllocationScore_s2> 

CTroopAllocation  locationld ="1"  allocation =”25" 
time_ms ="899"  /> 

<TroopAllocation  locationld^ ”2 ”  allocation =”25" 
time_ms= "400 "  /> 

<TroopAllocation  locationId= "3 "  allocation ="25" 
time_ms= "678 "  /> 

<TroopAllocation  locationld^ "4 "  allocation ="25" 
time_ms ="1780"  /> 

</TroopAllocationResponse> 

<GroundTruthSurpriseResponse  surprise^  "3  " 
trialPartTime_ms ="1840"  /> 

</TrialResponse> 

</Trial> 

</ExamPhase> 

<!--  Task  5  Response  --> 

<ExamPhase  xsirtyp e="ns3 : Task_5_Phase"  examld=  "Sample  Exam" 
name= "Mission  5"  id ="Task5" 
startTim e="2011-07-27T15 : 50 : 45 . 571-04 : 00" 
endTim e=" 2011-07 -27T16 : 20 : 45 . 571-04 : 00"> 

<Trial  trialNum  ="1"> 

<FeatureVectorFile  f eatureVectorUrl_CSV= "task5_l . csv" 
f eatureVectorUrl_KML= "task5_l . kml "  /> 

<RoadsFile  f  eatureVectorUrl  CS  \f=”  roads .  csv" 
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f eatureVectorUrl_KML= "roads . kml "  /> 

<RegionsFile  f eatureVectorUrl_CSV= " S0CINT_1 . csv" 
f eatureVectorUrl_KML= ”S0CINT_1 . csv"  /> 
<InitialHumintReport> 

<Groups>A  B  C  D</Groups> 

<Probabilities>25  25  25  25</Probabilities> 
</InitialHumintReport> 

<INTLayers> 

<INTLayer> 

<LayerType  xsi : type= "ns3 : Imin tLayer "  /> 

<GroupProbe> 

<AttackLocation  locationld ="1"  /> 

<Groups>A  B  C  D</Groups> 

</ GroupProbe> 

</INTLayer> 

<INTLayer> 

<LayerType  xsirtyp e= ”ns3 : MovintLayer”  /> 

<GroupProbe> 

<AttackLocation  locationld ="1"  /> 

<Groups>A  B  C  D</Groups> 

</ GroupProbe> 

</INTLayer> 

<INTLayer> 

<LayerType  xsi : type= "ns3 : SigintLayer"> 

<Group>B</ Group> 

</LayerType> 

<GroupProbe> 

<AttackLocation  locationld =”1”  /> 

<Groups>A  B  C  D</Groups> 

</ GroupProbe> 

</INTLayer> 

<INTLayer> 

<LayerType  xsi : type= ”ns3 : Socin tLayer "  /> 

<GroupProbe> 

<AttackLocation  locationld^ "1 "  /> 

<Groups>A  B  C  D</Groups> 

</ GroupProbe> 

</INTLayer> 

</INTLayers> 

<TroopAllocationProbe> 

<Groups>A  B  C  D</Groups> 

</TroopAllocationProbe> 

<GroundTruth  responsibleGroup= "C"  /> 

<GroundTruthSurpriseProbe  minSurprise= "0  " 
maxSurprise= " 6"  increment ="1"  /> 

<TrialResponse  trialTime_ms= ”15382 "> 

<ResponseFeedback  responseWellFormed= "true "> 

<GroundTruth  responsibleGroup= ”C"  /> 

<ProbabilitiesScore_sl>100 . 0</ProbabilitiesScore_sl> 

<TroopAllocationScore_s2>25 . 0</TroopAllocationScore_s2> 

</ResponseFeedback> 

<InitialProbabilities>25  25  25  25</InitialProbabilities> 
<GroupResponse_af terINT> 

<INTLayerShown  userSelected= "false "> 

<LayerType  xsirtyp e= "ns3 : ImintLayer"  /> 
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</INTLayerShown> 

<GroupResponse  locationId=  ”1 " 

trial Par tTime_ms= " 3783 "> 
<NormativeProbsCumulative> 

25.0  25.0  25.0  25.0 
</NormativeProbsCumulative> 
<ProbabilitiesScore_sl> 

100 . 0</ProbabilitiesScore_sl> 
<GroupAttackProbability  groups "A" 

probability^ "25 "  time_ms ="333"  /> 
<GroupAttackProbability  groups ”B ” 

probability =”25”  time_ms= ”899 ”  /> 
<GroupAttackProbability  groups ”C” 

probability^ ”25 ”  time_ms ="722"  /> 
<GroupAttackProbability  groups ”D” 

probability =”25"  time_ms= "455 ”  /> 
</ GroupResponse> 

</ GroupResponse_af terINT> 

<GroupResponse_af terINT> 

<INTLayerShown  userSelected= "false "> 

<LayerType  xsirtyp e= "ns3 :  MovintLayer"  /> 
</INTLayerShown> 

<GroupResponse  locationld ="1" 
trial Par tTime_ms= "3678 "> 
<NormativeProbsCumulative> 

25.0  25.0  25.0  25.0 
</NormativeProbsCumulative> 
<ProbabilitiesScore_sl> 

100 . 0</ProbabilitiesScore_sl> 
<GroupAttackProbability  groups "A" 

probability= "25 "  time_ms= "2 78 "  /> 
<GroupAttackProbability  groups "B " 

probability =”25"  time_ms= "1 928 "  /> 
<GroupAttackProbability  groups "C" 

probability^ ”25 "  time_ms= "703 "  /> 
<GroupAttackProbability  groups "D" 

probability =”25"  time_ms= "1 0 78 "  /> 
</ GroupResponse> 

</ GroupResponse_af terINT> 

<GroupResponse_af terINT> 

<INTLayerShown  userSelected= "false "> 

<LayerType  xsi : type= "ns3 : SlglntLayer"> 
<Group>B</Group> 

</LayerType> 

</INTLayerShown> 

<GroupResponse  locationld ="1" 
trial Par tTime_ms= "5788 "> 
<NormativeProbsCumulative> 

25.0  25.0  25.0  25.0 
</NormativeProbsCumulative> 
<ProbabilitiesScore_sl> 

100 . 0</ProbabilitiesScore_sl> 
<GroupAttackProbability  groups "A" 

probability^ "25 "  time_ms ="1078"  /> 
<GroupAttackProbability  groups "B " 

probability= "25 ”  time_ms= ”1 899 "  /> 
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<GroupAttackProbability  groups " C " 

probability ="25"  time_ms= " 762 ”  /> 
<GroupAttackProbability  groups "D" 

probability^  "25 "  time_ms ="563”  /> 
</GroupResponse> 

</ GroupResponse_af terINT> 

<GroupResponse_af terINT> 

<INTLayerShown  userSelected= "false "> 

<LayerType  xsi : type= "ns3 : Socin tLayer ”  /> 
</INTLayerShown> 

<GroupResponse  locationld ="1" 

trialPartTime_ms= "4598 ”> 
<NormativeProbsCumulative> 

25.0  25.0  25.0  25.0 
</NormativeProbsCumulative> 
<ProbabilitiesScore_sl> 

100 . 0</ProbabilitiesScore_sl> 
<GroupAttackProbability  groups "A" 

probability^ "25 "  time_ms ="1078”  /> 
<GroupAttackProbability  groups  "B " 

probability= "25 ”  time_ms= "344 "  /> 
<GroupAttackProbability  groups "C" 

probability= ”25 ”  time_ms= "98 7 ”  /> 
<GroupAttackProbability  groups "D" 

probability^ "25 "  time_ms ="432"  /> 
</ GroupResponse> 

</ GroupResponse_af terINT> 

<TroopAllocationResponse  trial Par tTime_ms= "8937 "> 

<T roopAl 1 o cationS co re_s 2 > 

25 . 0</TroopAllocationScore_s2> 
<TroopAllocation  groups "A"  allocation =”25" 
time_ms= ”453 "  /> 

<TroopAllocation  group ="B"  allocation =”25" 
time_ms ="2384"  /> 

<TroopAllocation  group ="C"  allocation ="25" 
time_ms ="1438"  /> 

<TroopAllocation  group ="D"  allocation ="25” 
time_ms= ”121 1 ”  /> 
</TroopAllocationResponse> 

<GroundTruthSurpriseResponse  surprise^ "3 " 
trialPartTime_ms= "2013 "  /> 

</TrialResponse> 

</Trial> 

</ExamPhase> 

<!--  Task  6  Response  --> 

<ExamPhase  xsi : type= "ns3 : Task_  6_Phase "  examld =" Sample  Exam" 
name= "Mission  6"  id ="Task6" 
startTim e=" 2011-07 -27T16 : 20 : 45 . 571-04  :  00" 
endTime=”2 01 1-07-27T1 6: 50: 45. 57 1-04: 00 "> 

CTrial  trialNum =”1"  numLayersToShow ="3”> 

<FeatureVectorFile  f eatureVectorUrl_CSV= ”task6_l . csv” 
f eatureVectorUrl_KML= ”task6_l . kml "  /> 

<RoadsFile  f eatureVectorUrl_CSV= "roads . csv" 
f eatureVectorUrl_KML= "roads . kml "  /> 

<RegionsFile  f eatureVectorUrl_CSV= ” SOCINT_l . csv" 
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f eatureVectorUrl_KML= "S0CINT_1 . csv"  /> 
<InitialHumintReport> 

<Groups>A  B  C  D</Groups> 

<Probabilities>25  25  25  25</Probabilities> 
</InitialHumintReport> 

<INTLayers> 

<INTLayer> 

<LayerType  xs i : type= "ns3 : ImintLayer"  /> 
<GroupProbe> 

<AttackLocation  locationId= ”3 "  /> 
<Groups>A  B  C  D</Groups> 

</ GroupProbe> 

</INTLayer> 

<INTLayer> 

<LayerType  xs i : type= ”ns3 : MovintLayer"  /> 
<GroupProbe> 

<AttackLocation  locationId= "3 "  /> 
<Groups>A  B  C  D</Groups> 

</ GroupProbe> 

</INTLayer> 

<INTLayer> 

<LayerType  xsi : type= ”ns3 : SigintLayer”> 
<Group>A</Group> 

</LayerType> 

<GroupProbe> 

<AttackLocation  locationId= ”3 "  /> 
<Groups>A  B  C  D</Groups> 

</ GroupProbe> 

</INTLayer> 

<INTLayer> 

<LayerType  xsi : type= ”ns3 : SigintLayer”> 
<Group>B</Group> 

</LayerType> 

<GroupProbe> 

<AttackLocation  locationld =”3”  /> 
<Groups>A  B  C  D</Groups> 

</ GroupProbe> 

</INTLayer> 

<INTLayer> 

<LayerType  xsi : type=  ”ns3 : SigintLayer”> 
<Group>C</Group> 

</LayerType> 

<GroupProbe> 

<AttackLocation  locationId= ”3 "  /> 
<Groups>A  B  C  D</Groups> 

</ GroupProbe> 

</INTLayer> 

<INTLayer> 

<LayerType  xsi : type= "ns3 : SigintLayer”> 
<Group>D</ Group> 

</LayerType> 

<GroupProbe> 

<AttackLocation  locationld^ "3 "  /> 
<Groups>A  B  C  D</Groups> 

</ GroupProbe> 

</INTLayer> 
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<INTLayer> 

<LayerType  xsirtyp e= "ns3 : SocintLayer"  /> 

<GroupProbe> 

<AttackLocation  locationId= "3 "  /> 

<Groups>A  B  C  D</Groups> 

</ GroupProbe> 

</INTLayer> 

</INTLayers> 

<TroopAllocationProbe> 

<Groups>A  B  C  D</Groups> 

</TroopAllocationProbe> 

<GroundTruth  responsibleGroup =”D”  /> 

<GroundTruthSurpriseProbe  minSurprise= ”0 ” 
maxSurprise=  ”6"  increment  =  "1"  /> 

<TrialResponse  trialTime_ms= " 664 96"> 

<ResponseFeedback  responseWellFormed= "true "> 

<GroundTruth  responsibleGroup^ "D"  /> 
<ProbabilitiesScore_sl> 

100 . 0</ProbabiiitiesScore_sl> 

<T roopAl 1 o cationS co re_s 2 > 

25 . 0</TroopAllocationScore_s2> 
</ResponseFeedback> 

<InitialProbabilities>25  25  25  25</InitialProbabilities> 
<GroupResponse_af terINT> 

<INTLayerExpectedUtilities> 

<ExpectedUtility 

expectedUtility_cumulative= "0 . 25 "> 
<LayerType  xsirtyp e= ”ns3 : ImintLayer"  /> 
</ExpectedUtility> 

<ExpectedUtility 

expectedUtility_cumulative= "0 . 25 "> 
<LayerType  xsirtyp e= ”ns3 : MovintLayer"  /> 
</ExpectedUtility> 

<ExpectedUtility 

expectedUtility_cumulative= " .25 "> 
<LayerType  xsi : type=  ”ns3 : SigintLayer"> 
<Group>A</Group> 

</LayerType> 

</ExpectedUtility> 

<ExpectedUtility 

expectedUtility_cumulative= "0 . 25 "> 
<LayerType  xsi : type= ”ns3 : SigintLayer"> 
<Group>B</Group> 

</LayerType> 

</ExpectedUtility> 

<ExpectedUtility 

expectedUtility_cumulative= "0 . 25 "> 
<LayerType  xsi : type= ”ns3 : SigintLayer"> 
<Group>C</ Group> 

</LayerType> 

</ExpectedUtility> 

<ExpectedUtility 

expectedUtility_cumulative= ”0 . 25 "> 
<LayerType  xsi r  type= "ns3 : SigintLayer"> 
<Group>D</ Group> 

</LayerType> 
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</ExpectedUtility> 

<ExpectedUtility 

expectedUtility_cumulative= " 0 . 25 ”> 
<LayerType  xsi  :  type=  ”ns3 :  Socin  tLayer  "  /> 
</ExpectedUtility> 

</INTLayerExpectedUtilities> 

<INTLayerShown  userSelected= " true " 
layer Select ionTime_ms= "23 12 "> 

<LayerType  xsi : type= "ns3 : SigintLayer"> 
<Group>A</ Group> 

</LayerType> 

</INTLayerShown> 

<GroupResponse  locationId= "1 " 

trialPartTime_ms= " 6030"> 
<NormativeProbsCumulative> 

25.0  25.0  25.0  25.0 
</NormativeProbsCumulative> 
<ProbabilitiesScore_sl> 

100 . 0</ProbabilitiesScore_sl> 
<GroupAttackProbability  groups "A" 

probability =”25"  time_ms= "1324 "  /> 
<GroupAttackProbability  groups "B " 

probability =”25"  time_ms= "1 123 "  /> 
<GroupAttackProbability  groups "C" 

probability^ "25 ”  time_ms ="1245"  /> 
<GroupAttackProbability  groups "D" 

probability =”25"  time_ms= "333 "  /> 

</ GroupResponse> 

</ GroupResponse_af terINT> 

<GroupResponse_af terINT> 

<INTLayerExpectedUtilities> 

<ExpectedUtility 

expectedUtility_cumulative= "0 . 25 "> 
<LayerType  xsi : type= "ns3 : Imin tLayer "  /> 
</ExpectedUtility> 

<ExpectedUtility 

expectedUtility_cumulative= ”0 . 25 "> 
<LayerType  xsi : type=  "ns3 :Movin tLayer "  /> 
</ExpectedUtility> 

<ExpectedUtility 

expectedUtility_cumulative= ”0 . 25 "> 
<LayerType  xsi : type= ”ns3 : Socin tLayer "  /> 
</ExpectedUtility> 

</INTLayerExpectedUtilities> 

<INTLayerShown  userSelected= "true " 
layerSelectionTime_ms= "1 638 "> 

<LayerType  xsi : type= "ns3 :Movin tLayer "  /> 
</INTLayerShown> 

<GroupResponse  locationld ="1" 

trial Par tTime_ms= "12075 "> 
<NormativeProbsCumulative> 

25.0  25.0  25.0  25.0 
</NormativeProbsCumulative> 
<ProbabilitiesScore_sl> 

100 . 0</ProbabilitiesScore_sl> 
<GroupAttackProbability  groups "A" 
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probability^ "25 "  time_ms= "30 78 "  /> 
<GroupAttackProbability  groups "B " 

probability= "25 "  time_ms= "3434 "  /> 
<GroupAttackProbability  groups "C" 

probability  =”25"  time_ms  ="1234,r  /> 
<GroupAttackProbability  groups "D" 

probability= "25 "  time_ms ="4532"  /> 

</ GroupResponse> 

</ GroupResponse_af terINT> 

<GroupResponse_af terINT> 

<INTLayerExpectedUtilities> 

<ExpectedUtility 

expectedUtility_cumulative= "0 . 25 "> 
<LayerType  xs i : type= "ns3 : ImintLayer"  /> 
</ExpectedUtility> 

<ExpectedUtility 

expectedUtility_cumulative= ”0 . 25 "> 
<LayerType  xsirtyp e=  "fls3  :  SocintLayer"  /> 
</ExpectedUtility> 

</INTLayerExpectedUtilities> 

<INTLayerShown  userSelected= "true " 
layerSelectionTime_ms= "6084 "> 

<LayerType  xsi : type= "ns3 : Imin tLayer "  /> 
</INTLayerShown> 

<GroupResponse  locationld ="1" 
trial Par tTime_ms= "4213"> 
<NormativeProbsCumulative> 

25.0  25.0  25.0  25.0 
</NormativeProbsCumulative> 
<ProbabilitiesScore_sl> 

100 . 0</ProbabiiitiesScore_sl> 
<GroupAttackProbability  groups "A" 

probability= "25 "  time_ms= "783 "  /> 
<GroupAttackProbability  groups "B " 

probability^ "25 "  time_ms ="1323"  /> 
<GroupAttackProbability  groups "C" 

probability= "25 "  time_ms ="346"  /> 
<GroupAttackProbability  groups "D" 

probability^ "25 "  time_ms ="467"  /> 
</GroupResponse> 

</ GroupResponse_af terINT> 

<TroopAllocationResponse  trialPartTime_ms= "7854 ”> 

<T roopAl 1 o cationS co re_s 2 > 

25 . 0</TroopAllocationScore_s2> 

<TroopAllocation  groups "A"  allocation ="25" 
time_ms ="345"  /> 

<TroopAllocation  groups "B"  allocation= "25 " 
time_ms= "68 7"  /> 

<TroopAllocation  group ="C"  allocation ="25" 
time_ms ="5677"  /> 

<TroopAllocation  group ="D"  allocation= ”25 " 
time_ms= ”456 "  /> 

</TroopAllocationResponse> 

<GroundTruthSurpriseResponse  surprise^ "3 ”  /> 
</TrialResponse> 

</Trial> 
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</ExamPhase> 

<!--  Task  7  --> 

<ExamPhase  xs i : type= "ns3 : Task_ 7_Phase "  examld= "Sample  Exam" 
name= "Mission  7"  id= " Task  7 " 
startTim  e="2011-08-HT12 : 37 : 51 . 054-04:00" 
endTim  e="2011-08-HT13 :  07 : 51 . 054-04:00"> 

<RoadsFile  f eatureVectorUrl_CSV= "roads . csv" 
f eatureVectorUrl_KML= "roads . kml ”  /> 

<RegionsFile  f eatureVectorUrl_CSV= "S0CINT_1 . csv" 
f eatureVectorUrl_KML= ” S0CINT_1 . csv"  /> 
<InitialResponsibleGroup>A</InitialResponsibleGroup> 
<InitialCredits>10</InitialCredits> 

<CorrectPredictionCredits>l</ CorrectPredictionCredits> 

<Trial  trialNum  ="1"> 

<FeatureVectorFile  f eatureVectorUrl_CSV= "task7_l . csv" 
f eatureVectorUrl_KML= "task 7 _1 . kml "  /> 

<GroupProbe> 

<Groups>A  B  C  D</Groups> 

</ GroupProbe> 

<LocationProbe> 

<Locations>l  2  3  4</Locations> 

</LocationProbe> 

<TroopAllocationProbe> 

<Locations>l  2  3  4</Locations> 
</TroopAllocationProbe> 

<ResponsibleGroupShown>true</ResponsibleGroupShown> 
<GroundTruth  responsibleGroup =”B”  attackLocationld ="4"  /> 
<INTLayers> 

<INTLayer  costCredits =”1"> 

<LayerType  xsirtyp e= ”ns3 : ImintLayer"  /> 
</INTLayer> 

<INTLayer  costCredits ="1"> 

<LayerType  xsi : type=  "ns3 :Movin tLayer "  /> 
</INTLayer> 

<INTLayer  costCredits^ "2 "> 

<LayerType  xsi : type= "ns3 : SigintLayer"> 
<Group>A</Group> 

</LayerType> 

</INTLayer> 

<INTLayer  costCredits^  "2 "> 

<LayerType  xsi : type= "ns3 : SigintLayer"> 
<Group>B</Group> 

</LayerType> 

</INTLayer> 

<INTLayer  costCredits^ "2 "> 

<LayerType  xsi : type=  "ns3 : SigintLayer"> 
<Group>C</Group> 

</LayerType> 

</INTLayer> 

<INTLayer  costCredits^ ”2 "> 

<LayerType  xsi : type=  ”ns3 : SigintLayer"> 
<Group>D</Group> 

</LayerType> 

</INTLayer> 

<INTLayer  costCredits^ ”2 "> 
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<LayerType  xsiityp e= "ns3 : SocintLayer"  /> 
</INTLayer> 

</INTLayers> 

<TrialResponse  trialTime_ms= "45187 "> 

<ResponseFeedback  responseWellFormed= "true "> 
<GroundTruth  responsibleGroup= "B " 
attackLocationId= ”4 "  /> 
<NumCreditsAwarded>0 . 25</NumCreditsAwarded> 
<NumCreditsRemaining>7 . 25</NumCreditsRemaining> 
<NumCreditsUsed>3 . 0</NumCreditsUsed> 

<T roopAl 1 o cationS co re_s 2 > 

25 . 0</TroopAllocationScore_s2> 
</ResponseFeedback> 

<GroupResponse  trialPartTime_ms= "33542 "> 
<GroupAttackProbability  groups "A" 
probability= ”85 "  /> 
<GroupAttackProbability  group= ”B " 
probability= ”5 "  /> 

<GroupAttackProbability  groups "C" 
probability^ "5 "  /> 

<GroupAttackProbability  group= "D" 
probability= "5 "  /> 

</ GroupResponse> 

<LocationResponse  trialPartTime_ms= ” 6243"> 

<GroupAttackProbability  locationld ="1 " 
probability=  "25 "  time_ms =”456"  /> 
<GroupAttackProbability  locationId= ”2 " 
probability=  ”25 "  time_ms =”864"  /> 
<GroupAttackProbability  locationId= "3 " 

probability^ "25 "  time_ms ="1783"  /> 
<GroupAttackProbability  locationId= "4 " 

probability= ”25 "  time_ms= "2456 "  /> 
</LocationResponse> 

<TroopAllocationResponse  trialPartTime_ms= "3433 "> 

<T roopAl 1 o cationS co re_s 2 > 

25 . 0</TroopAllocationScore_s2> 
<TroopAllocation  locationld ="1"  allocation ="25" 
time_ms= "325 "  /> 

<TroopAllocation  locationld^ ”2 "  allocation ="25" 
time_ms= ”673 ”  /> 

<TroopAllocation  locationld^ "3 "  allocation= "25 " 
time_ms= ”43 7 "  /> 

CTroopAllocation  locationld^ "4 "  allocation ="25" 
time_ms= "855 "  /> 
</TroopAllocationResponse> 

<LayerPurchaseTime_ms>57 18</ Layer Pur chaseTime_ms> 
<INTLayerPurchase  costCredits ="1 "> 

<INTLayer  xsiityp e= "ns3 : ImintLayer"  /> 
</INTLayerPurchase> 

<INTLayer Pur chase  costCredits= "2 ”> 

<INTLayer  xsi : type= "ns3 : SigintLayer"> 
<Group>C</Group> 

</INTLayer> 

</INTLayerPurchase> 

</TrialResponse> 

</Trial> 
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CTrial  trialNum= "2 "> 

<FeatureVec tor File  f eatureVectorUrl_CSV= ” task  7 _2 . csv" 
f eatureVectorUrl_KML= "task 1_2 . kml "  /> 

<GroupProbe> 

<Groups>A  B  C  D</Groups> 

</ GroupProbe> 

<LocationProbe> 

<Locations>l  2  3  4</Locations> 

</LocationProbe> 

<TroopAllocationProbe> 

<Locations>l  2  3  4</Locations> 

</TroopAllocationProbe> 

<ResponsibleGroupShown>true</ResponsibleGroupShown> 
<GroundTruth  responsibleGroup= "A "  attackLocationld ="1"  /> 
<TrialResponse  trialTime_ms= "45187 "> 

<ResponseFeedback  responseWellFormed= "true "> 
<GroundTruth  responsibleGroup= "A" 
attackLocationld ="1"  /> 
<NumCreditsAwarded>0 . 25</NumCreditsAwarded> 
<NumCreditsRemaining>7 . 5</NumCreditsRemaining> 
<NumCreditsUsed>0 . 0</NumCreditsUsed> 

<T roopAl 1 o cationS co re_s 2 > 

25 . 0</TroopAllocationScore_s2> 
</ResponseFeedback> 

<GroupResponse  trialPartTime_ms= "33542 "> 
<GroupAttackProbability  group= "A" 

probability= ”90 ”  time_ms =”1238"  /> 
<GroupAttackProbability  group= "B " 

probability^  "2 "  time_ms ="4567"  /> 
<GroupAttackProbability  groups "C" 

probability= "2 ”  time_ms =”1245"  /> 
<GroupAttackProbability  group= "D" 

probability= " 6”  time_ms =”2356"  /> 
</GroupResponse> 

<LocationResponse  trialPartTime_ms= "6243"> 

<GroupAttackProbability  locationId= ”1 " 
probability= ”25 "  time_ms= "45 6"  /> 
<GroupAttackProbability  locationId= ”2 " 
probability^ "25 "  time_ms= "864 "  /> 
<GroupAttackProbability  locationId=  "3 " 

probability= ”25 "  time_ms =”1783"  /> 
<GroupAttackProbability  locationId= "4 " 

probability= ”25 "  time_ms ="2456"  /> 
</LocationResponse> 

<TroopAllocationResponse  trialPartTime_ms= "3433 "> 

<T roopAl 1 o cationS co re_s 2 > 

25 . 0</TroopAllocationScore_s2> 
<TroopAllocation  locationld ="1"  allocation ="25" 
time_ms ="325"  /> 

CTroopAllocation  locationId= "2 "  allocation ="25" 
time_ms= "673 "  /> 

CTroopAllocation  locationId= "3 "  allocation ="25" 
time_ms ="437"  /> 

CTroopAllocation  locationld ="4"  allocation ="25" 
time_ms= "855 "  /> 
c/TroopAllocationResponse> 
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</TrialResponse> 

</Trial> 

</ExamPhase> 

</ns3 : IcarusEvaluation  CPD1> 
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